Healthcheck for NetworkLoadBalancer with UDP ECS service

Question

I'm trying to deploy an ECS service that uses only a UDP port. Support has been added for UDP load-balancing via NetworkLoadBalancers, so I've deployed my service allowing dynamic host port assignment for my tasks and setup the NLB with an appropriate listener and target group.

The problem I'm running into is that healthchecks are apparently mandatory for the NLB, and must be TCP based. For the healthcheck port, you can leave the default "target" port (which works fine for dynamic host port assignment) or you can specify a port. What I can't do is have a different port exposed for TCP than my load balancer target UDP port. I could have my container listen to both UDP for production and TCP for healthchecks on the same port, but the task definition seems to disallow that even though Docker supports it fine.

This would seem to make UDP NLB's useless for ECS services, unless there's something I'm missing? The only alternative I've come up with is to use statically configured host ports so I can expose a second port for TCP on a controlled host port and target that for the NLB healthcheck. The problem with that is we've now lost all of the scalability benefit of ECS by not being able to run more than one task on an instance.

Answer 1

What you can do is setup a sidecar container along side your UDP container that supplies the TCP endpoint for health checks.

Here is a truncated example of the ECS Task Definition for the service that is running in our NLB target group:

{
    "containerDefinitions": [
        {
            "image": "[your-udp-image]",
            "essential": true,
            "portMappings": [
                {
                    "containerPort": 5008,
                    "protocol":"udp"
                }
            ]
        },
        {
            "image": "[your-tcp-health-check-image]",
            "essential": true,
            "portMappings": [
                {
                    "containerPort": 5006,
                    "protocol":"tcp"
                }
            ],
            "healthCheck": {
                "command": [ "CMD-SHELL", "curl -f http://localhost:5006 || exit 1" ],
                "interval": 10,
                "timeout": 5,
                "retries": 3,
                "startPeriod": 120
            }
        }
    ]
}

Then your target group's health check settings can just point to the TCP path and port of your health check container.