Prevent rollback in AWS cloudformation

Question

Currently, my CFN is in update_rollback_failed state and apparently its a dead end state. To continue, I have to continue the rollback which is not an option.

We pushed a CFN 3 months back and due to some issue we made some changes directly from the console (huge mistake) and now we were trying to sync what we changed in the CFN itself. Due to some issue the deployment failed and it started doing rollback. Luckily, the rollback failed because of some checks on kinesis side.

The CFN available on AWS contains old config which is catastrophic and would definitely lead to some significant downtime.

I have tried continuing the rollback ignoring the resources but it allows to ignore only those resources for which the rollback update failed. How should I get out of this situation?

Answer 1

Unfortunately, this is a known limitation of CloudFormation and there is no work around to it. There is an AWS blog post that mentions update_rollback_failed that confirms that this is something we can't do anything about, it says the following:

When a stack reaches UPDATE_ROLLBACK_FAILED, this means that the CloudFormation stack was attempting an UPDATE operation, the operation failed, and we began a rollback. An issue occurred that stopped CloudFormation from returning to the previous “good” state during the rollback. As a result, the stack can’t update and can’t roll back, thus it assumes this half-way state. The API then stops any further actions on the stack other than ContinueUpdateRollback and DeleteStack.