App engine instances spikes

Question

I am using automatic scaling with GAE, running php55. This also happened with php5. Settings:

automatic_scaling:
  min_idle_instances: 0
  max_idle_instances: 2  # default value
  min_pending_latency: 500ms
  max_pending_latency: 7000ms

I am having trouble understanding why these spikes are happening. Typically my application requires no more than 5 instances running at once. Occasionally this will jump to 1,200+ for no apparent reason. The logs surrounding this time show 500 timeout errors for all non-static content.

The only dependency on these pages is a simple database insert. I am using Google Cloud SQL. There are no errors reported in the Cloud SQL logs either.

Any ideas on how to further troubleshoot this? As you can see from the image, this problem is very sporadic but extremely costly.

Answer 1

This was caused because the database shut down after a short while of inactivity. When traffic came back it took time restart the database server. During this time, app engine thought it needed to create more instances to keep up with the heavy load.

I solved this by leaving my database instance 'always on'.