ElasticSearch on Elastic Beanstalk

Question

I'm trying to get ElasticSearch running in an Elastic Beanstalk environment. Using Docker image it's fairly straightforward to get one instance running in a load balanced environment. However, when I try to add more instances to the cluster, they fail to discover each other and every new one becomes a new_master.

My Dockerfile looks like following

FROM dockerfile/java:oracle-java8
RUN ... # Downloading and installing ElasticSearch
RUN /elasticsearch/bin/plugin install elasticsearch/elasticsearch-cloud-aws/2.5.0
VOLUME ["/data"]
ADD config/elasticsearch.yml /elasticsearch/config/elasticsearch.yml
WORKDIR /data
CMD ["/elasticsearch/bin/elasticsearch"]

EXPOSE 9200

And the configuration config/elasticsearch.yml looks like following:

cluster:
  name: elastic-env-dev
cloud:
  aws:
    region: ap-southeast-2
discovery:
  type: ec2
  ec2:
    tag:
      Name: elastic-env-dev
    ping_timeout: 120s

The name of the EB environment is elastic-env-dev.

Answer 1

1) Check your instance security groups, in order for ES instances to talk to each other they must use port 9300

2) AWS does not allow multicast. you must disable multicast in the elasticsearch config

add this line to the config on each ES config

discovery.zen.ping.multicast.enabled: false

if that doesn't work, try adding unicast config

discovery.zen.ping.unicast.hosts: loadbalancer.address

3) remember the security groups on instances must allow 9300 from the ELB

Custom TCP Rule  TCP  9300 amazon-elb/sg-123456ed (amazon-elb-sg)

4) use telnet to check comm between the ES instances by using

telnet ip_address 9300

Answer 2

It is possible to deploy ES on EBT, here is how

http://vladmiller.com/elasticsearch/aws/2015/12/08/deploy-elasticsearch-on-aws-elasticbeanstalk.html

---

At this point (November 14 2015), after spend some good amount of time I'll say that it's not possible to make ES cluster works on EB.

Problem #1 is that you have to map docker port to host machine as if you would do

docker run -p 9300:9300 ...

This could be very easily solved if you add post appdeploy hook via .ebextensions which will setup iptables port forwarding

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_setup_iptables.sh":
    mode: "0755"
    owner: root
    group: root
    content: |
      #!/bin/sh
      iptables-save | grep -v added_by_ebextension | iptables-restore

      DOCKER_IP=$(docker inspect `cat /etc/elasticbeanstalk/.aws_beanstalk.current-container-id` | jq -r .[0].NetworkSettings.IPAddress)
      iptables -t nat -A DOCKER -p tcp --dport 9300:9400 -j DNAT --to-destination ${DOCKER_IP} -m comment --comment added_by_ebextension

      service iptables save

Problem #2 you need to adjust security group, make sure you allow TCP 9300-9400 and ICMP traffic between nodes in SG.

Problem #3 use aws-ec2 discovery plugin and limit it to your SG, therefore no other machines get's discovered

// elasticsearch.yml

cloud.aws:
  access_key: YYYYYYYYY
  secret_key: XXXXXXXXX
  region: us-east-1

discovery.type: ec2
discovery.ec2.ping_timeout: 30s
discovery.ec2.tag.Name: [ENVIRONMENT_NAME]
discovery.ec2.host_type: private_dns
discovery.zen.ping.multicast.enabled: false

Problem #4, unsolved is that each ES node will bind to internal docker IP address, which is something like 172.17.0.3, however your host private IP is different. Therefore when nodes will discover each other and start to communicate they will report wrong IP address to others.

[2015-11-13 21:50:58,542][TRACE][discovery.zen.ping.unicast] [86ac0ad55d5b] [2] received response from {#zen_unicast_21_#cloud-i-8c317a3b-0#}{10.165.71.177}{ip-10-165-71-177.ec2.internal/10.165.71.177:9300}: [ping_response{node [{86ac0ad55d5b}{U3PF5qOaQCucpK3JfZ3ARA}{172.17.0.3}{172.17.0.3:9300}], id[5], master [null], hasJoinedOnce [false], cluster_name[es-staging]}, ping_response{node [{86ac0ad55d5b}{U3PF5qOaQCucpK3JfZ3ARA}{172.17.0.3}{172.17.0.3:9300}], id[7], master [null], hasJoinedOnce [false], cluster_name[es-staging]}, ping_response{node [{86ac0ad55d5b}{U3PF5qOaQCucpK3JfZ3ARA}{172.17.0.3}{172.17.0.3:9300}], id[9], master [null], hasJoinedOnce [false], cluster_name[es-staging]}, ping_response{node [{86ac0ad55d5b}{U3PF5qOaQCucpK3JfZ3ARA}{172.17.0.3}{172.17.0.3:9300}], id[11], master [null], hasJoinedOnce [false], cluster_name[es-staging]}, ping_response{node [{89764d1bb185}{yVRC-HmIQoayIuWfi6a09g}{172.17.0.3}{172.17.0.3:9300}], id[30], master [{89764d1bb185}{yVRC-HmIQoayIuWfi6a09g}{172.17.0.3}{172.17.0.3:9300}], hasJoinedOnce [true], cluster_name[es-staging]}]

You can see that node discovered on ip-10-165-71-177.ec2.internal/10.165.71.177:9300, however that node responded that it's IP is 172.17.0.3 thus first node instead of connection to EC2 private IP will try to connect to internal Docker IP

[2015-11-13 21:51:00,037][TRACE][discovery.ec2 ] [86ac0ad55d5b] full ping responses:
--> ping_response{node [{89764d1bb185}{yVRC-HmIQoayIuWfi6a09g}{172.17.0.3}{172.17.0.3:9300}], id[30], master [{89764d1bb185}{yVRC-HmIQoayIuWfi6a09g}{172.17.0.3}{172.17.0.3:9300}], hasJoinedOnce [true], cluster_name[es-staging]}
[2015-11-13 21:51:00,041][DEBUG][discovery.ec2 ] [86ac0ad55d5b] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [{89764d1bb185}{yVRC-HmIQoayIuWfi6a09g}{172.17.0.3}{172.17.0.3:9300}], id[30], master [{89764d1bb185}{yVRC-HmIQoayIuWfi6a09g}{172.17.0.3}{172.17.0.3:9300}], hasJoinedOnce [true], cluster_name[es-staging]}

We need to somehow make ES to bind to host's IP address or ignore those docker IP addresses and continue on discovered IP.

---

UPDATE 1 I suspect that you can deploy ES to EB without Docker, however I have not tried this option yet.

---

UPDATE 2 I was able to make nodes discover each other and try to communicate, however now it has a different issue

---

UPDATE 3 Here is the story and sample code on how to achieve desired effect http://vladmiller.com/elasticsearch/aws/2015/12/08/deploy-elasticsearch-on-aws-elasticbeanstalk.html