Elastic Beanstalk Environment stuck on grey health

Question

My AWS Elastic Beanstalk Environment is stuck in Health: Grey.

My application is working, I can access it fine. However, I am unable to change configuration or deploy new versions because I get a message saying that

Environment named ______ is in an invalid state for this operation. Mustbe Ready.

If I run eb health on my console, I get the following output:

Status: Ready Health Grey

And

ELB State: InService

Is there anything I can try to revive my environment? I have contacted AWS Support, by they are really slow. Another option I can think of is terminating the environment and creating a new one, but I really would prefer to avoid that.

Answer 1

EB can be fairly tough to trouble shoot when you have full access to EB, the instances, the ELB's, etc... never mind trying to proxy this through SO.

I'd do the following:

• Bring up a new environment under the same application
• When it comes up green, use the EB application "Swap" functionality to swap the environments

More details on this process is here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.CNAMESwap.html

This performs a DNS switch so you should have no down time. You'll still the old environment running if you want to troublshoot it later with your friendly AWS support staff.

The only negatives are:

• You'll continue paying for both environment stacks while waiting to troubleshoot the other.
• The DNS is a little tough as you can't guarantee clients respect the short time-outs EB DNS entries have. They should, but someone may decide to keep using a local cached version. As with anything relying on trusting client-side features, it's a bit out of your control.
• If you deploy a RDS DB via EB, you can't us the swap as the DB is tied to the environment (NEVER deploy a RDS DB in production EB environment via EB!!!!)

Answer 2

I know the question's already been answered, but I think the cause of the problem is important, instead of recommending a complete rebuild of OP's environment.

Elastic Beanstalk has 4 different colors - green, yellow, red, and grey. However, each color can mean multiple different things that vary wildly. Here are the potential statuses behind the grey color:

Grey (Suspended) - Your application has had such severe health issues Elastic Beanstalk is no longer monitoring it

Grey (Unknown) - The health agent has not reported enough data on an instance yet

Grey (Pending) - An operation is in progress on an instance within the command timeout (for example bootstrapping the environment)

Notice the incredible disparity between "Pending" and "Suspended". In Pending, it just needs a little more time, or perhaps you can shut down a single resource and have it respawn. In Suspended, all monitoring is shot, and you ought to rebuild the environment ASAP. Big difference in impact to customers during the solution.

Baked into Beanstalk are the vanilla colors. To get the additional statuses, you have to enable Enhanced Monitoring. You can do it in a couple minutes, and the cost is nominal.

To read more about the statuses and common problems with Beanstalk, I'd recommend a blog my colleague wrote: Health Monitoring in AWS Beanstalk