Linking containers between task definitions in AWS ECS?

Question

I'm trying to setup a basic web application, which has an associated database, in AWS ECS. Locally I have these setup in different containers, and on ECS, I'd like to have separate task definitions so that I may scale the two separately.

1. I registered my first task definition as david_mongodb successfully in ECS. It has a container named david_mongodb in it.
2. Then I attempted to register my second task definition as david_web, which has a container named david_web that links the database via david_mongodb:db.

3. When I click 'Create', it returns an error:

   Unable to create Task Definition Linked container 'david_mongodb:db' doesn't exist. 

It seems like task definitions can't see container names in other task definitions? I'm thinking putting both david_web and david_mongodb containers in the same task definition would work, but I don't want to do that: it would prevent me from scaling either web app or database separately. This overview seems to confirm that my architecture is recommended...

So how do I link containers that live in different task definitions? Or is there another clever way of handling this?

Answer 1

Links in an ECS task definition are analogous to Docker links and only work when the containers are part of the same task definition (containers that are part of a single task definition are placed together on the same host). In order to communicate between containers in different task definitions, you'll need a mechanism for discovering where the containers are located (what host) as well as the port for communication.

ECS has integration with Elastic Load Balancing (Application Load Balancers, Network Load Balancers, and Classic Load Balancers) through the service feature, where tasks will be automatically registered in the ELB and deregistered in the ELB appropriately.

ECS also has integration with Route 53 Auto Naming for DNS-based service discovery using A and SRV records. Your service's tasks can be automatically entered into and removed from DNS records.

Service Discovery for Amazon ECS Using DNS describes a different approach where a Lambda function listens to the ECS event stream through CloudWatch Events and updates Route 53 DNS records. This method has been superceded by the Route 53 Auto Naming feature described above.

If you want to avoid load balancers and DNS, another pattern might be an ambassador container (there's a sample called the ecs-task-kite that uses the ECS API) or you might be interested in an overlay network (Weave has a fairly detailed getting started guide for their solution).

Nathan Peck is keeping track of a number of different subjects related to ECS, including service discovery, here.