I've developed a Docker based application comprised of multiple microservices. It has to consume Amazon SQS messages and processes them. At first I wanted to use AWS Elastic Beanstalk, but then I fell over the EC2 Container Service. Now I don't know which one to choose.
As of now, Elastic Beanstalk supports Multi-Container-Environments. That's great because every microservice has its own application server inside a docker container. The next problem is scaling:
I don't know how the scaling mechanism works. For example: I have 5 docker containers in my Elastic Beanstalk Environment. Now only the fifth docker container is under heavy load, because it has a huge amount of SQS messages to process, the other four are nearly idle, because they don't need much CPU or maybe don't have a lot of SQS messages. Let's assume the 5th container runs a JBoss application server. As far as i know, the server only can consume a limited amount of parallel requests even if there is enough CPU/memory available.
If the JBoss Docker container isn't able to handle the amount of requests, but there is enough CPU/memory available, of course I want to automatically start a second Docker/JBoss container on the same instance. But what happens, if I don't have enough CPU/memory? Of course I want to spin on a second instance, which is configurable through a auto-scaling group in EB. Now a second instance spins up, but every container except of the 5th is nearly idle, of course I don't want them to spawn 4 unnecessary at the second instance too, which would be a waste of resources. Only the 5th should spawn and the others should scale like the 5th scale based on configurable parameters like e.g.: CPU/memory/SQS.
I don't exactly know if Amazon ECS is doing that, or if it's possible at all, but I really can't find any source on the internet about this topic, which is in general said, scaling based on instances/containers.
ECS helps in having a more fine-grained control for custom application architectures. Elastic Beanstalk is ideal to leverage the benefits of containers but just want the simplicity of deploying applications from development to production by uploading a container image.
Amazon ECS provides tools to manage a cluster of instances running Docker containers. Elastic Beanstalk takes care of Amazon ECS tasks including cluster creation, task definition and execution. Each of the instances in the environment run the same set of containers, which are defined in a Dockerrun.
In the case that you want to reduce system operations and just focus on the website, then Elastic Beanstalk would be the best choice for that. Elastic Beanstalk supports a PHP stack (as well as others). You can keep your site in version control and easily deploy to your environment whenever you make changes.
Amazon Elastic Container Service (Amazon ECS) is a highly scalable and fast container management service. You can use it to run, stop, and manage containers on a cluster.
Deploying Docker containers directly from an Ec2 instance. Using Docker containers on Elastic Beanstalk. Docker cluster management using the AWS EC2 Container Service. Docker can run anywhere, on a racked server, an old laptop, and perhaps, if you worked at it hard enough, even on a smartphone.
Q: How is Amazon ECS different from AWS Elastic Beanstalk? AWS Elastic Beanstalk is an application management platform that helps customers easily deploy and scale web applications and services.
Q: What is Amazon Elastic Container Service? Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.
Since “Amazon” and “scale” are like two sides of the same coin, it wasn’t much of a surprise when they created the AWS EC2 Container Service to handle the installation, operation, scaling and general cluster management for you. Container instances communicate with the ECS service through an “ECS agent.”
EB vs ECS really comes down to control. Do you want to control your scaling and capacity or do you want to have that more abstracted and instead focus primarily on your app. ECS will give you control, as you have to specify the size and number of nodes in the cluster and whether or not auto-scaling should be used. With EB, you simply provide a Dockerfile and EB takes care of scaling your provisioning of number and size of nodes, you basically can forget about the infrastructure with the EB route.
Here's the EB documentation on Docker: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html
With ECS you'll have to build the infrastructure first before you can start deploying the the Dockerfile so it really comes down to 1) your familiarity with infrastructure and 2) level of effort that you want to spend on the infrastructure vs the app.
Not to resurrect a dead question, but hopefully this helps someone.
The accepted answer is not clear enough: based on what OP described, OP wants ECS, not Multi-Container Elastic Beanstalk (MCEB). As far as I can tell, MCEB never attempts to efficiently pack containers into instances. OP asks in a comment, "if only one is under load, it scales only this one, or does it always scale up the instances and start all containers, no matter under what load they are ?" And the answer is "the latter"; MCEB scales up the instances and starts all containers, no matter what load they are under.
Edit
Don't use the architecture you're imagining.
How micro are your microservices? Would it be ridiculous to give them each a t2.nano? Then make them each a single-container Docker EB app - EB worker applications can be driven by SQS messages. Or use apex.run.
Edit 1/31/18:
AWS Fargate seems pretty cool.
Edit 6/5/19:
Use EKS if you need to orchestrate containers, to satisfy an itch. But really, try to avoid this. Distributed systems are hard.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With