Elastic Search on AWS Fargate

Question

Facing a problem while deploying elasticsearch on AWS Fargate

Following steps were done :

customized my docker image and pushed to AWS ECR. task definition for my elasticsearch service

elastic search service fails on bootstrap following is the exception [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

its known es issue for es 5.0 onwards. Solution provided by es is as follows sysctl -w vm.max_map_count=262144

https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode

is it possible to apply this command on AWS Fargate as we donot have access to host ?

Update : Elastic Search has provided an option to avoid mmaps check on bootup but not yet released as of now

https://github.com/elastic/elasticsearch/pull/32421

https://discuss.elastic.co/t/elk-on-aws-fargate/153967/4

Answer 1

It looks like you cannot do that.

Let me explain:

Docker actually wraps a process and runs it using the kernel installed on the host machine.

Changing "vm.max_map_count" is actually configuring the Linux kernel of the host machine.

When the host machine is under your control, such as when you use EC2, you can configure the kernel of the host machine by applying "user data" on your Launch Configuration. (See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bootstrap_container_instance.html)

But where host machine is not under your control, as in the case of Fargate, you cannot change the host and the kernel settings it runs. The whole idea of Fargate is to run stateless Docker images, images that don't make any assumptions on the host they run inside.

However, in Elasticsearch, the application itself depends on specific host configuration (the "vm.max_map_count" setting) which means that it indeed does make assumptions about its host and therefore it cannot run on generic host such as Fargate (unless you disable this check, which is not a good idea for production environment).