Scaling ECS EC2 instances when a task cannot be placed

Question

I am using an ECS cluster for Jenkins agents/slaves with the Jenkins ECS plugin.

The plugin places a ECS Task when a job requests a build-node. Now I want to scale the EC2 instances in a Autoscaling Group associated with the ECS Cluster according to the demand.

1. The jenkins is often idle. In this case, I do not want there to be any instances in the autoscaling group.
2. If a node (and therefore an ECS task) is requested and cannot be placed, I want to add an EC2 instance to the autoscaling group.
3. If an instance is idle and shortly before an billing hour, I want that instance to be removed.

The 3. point can be accomplished by a cronjob on the EC2 instances that regularly checks if the conditions are met and removes the EC2 instance.

But how can I accomplish the 2. point? I am unable to create a cloudwatch alarm that triggers, if a task cannot be placed.

How can I accomplish this?

Answer 1

A rather hacky way to achieve this: You could use a Lambda function to detect when a service has runningCount + pendingCount < desiredCount for more than X seconds. (I have not tested this yet.) Similar solutions are proposed here.

There does not seem to be a proper solution to scale only when tasks cannot be placed. Maybe AWS wants us to over-provision our clusters, which might be good practice for high availability, but not always the best or cheapest solution.