Force terminate AutoScaling instances stuck in Pending:Wait

Question

I have an AutoScaling group where I'm messing around with LifecycleHooks on instance start, some controlled by me and some controlled by other AWS services. I've been adding/removing LifecycleHooks and changing the size of the ASG, and instances are getting stuck in Pending:Wait. I assume it's because it's waiting for a signal from a LifecycleHook, but I can't get it to budge.

I tried a number of things:

Terminate instances in EC2 console

I terminated the instances, but ASG waits for the LifecycleHook heartbeat to timeout before actually terminating the instance from the ASG, which is probably an hour.

Complete LifecycleHooks manually

I tried to complete the LifecycleHooks manually, illustrated by the following pseudocode:

describeAutoScalingGroups -> asg {
    instances = getPending(asg.instances)
    describeLifecycleHooks -> lifecycleHooks {
        lifecycleHooks.each {
            instances.each {
                completeLifecycleAction(instance, hook)
            }
        }
    }
}

This doesn't do the trick. I'm guessing the LifecycleHook it's waiting on was deleted from the ASG, so there's no way to manually complete the LifecycleHook.

What next?

Obviously, I should be more careful about deleting these resources in the right order and all that and decreasing the heartbeat will help as well, but how can I force terminate instances from an ASG no matter what it's waiting for?

Answer 1

After terminating EC2 instance in Console, manually completing LifecycleHook worked for me:

aws autoscaling complete-lifecycle-action --lifecycle-hook-name $HOOK --auto-scaling-group-name $ASG --lifecycle-action-result ABANDON --instance-id $ID