Suddenly huge number of instances

Question

we have an application running with instance type F1. It was running fine for more than a month now, idle instances and delay set to fully automatic. It never used more than one instance. Today, without any changes or deployments, we suddenly have 73 instances up, each with very few requests per instance (eg, 1 to 20). Naturally, we ran out of credits fast. Any idea what is going on here? I was also starting to play with the scheduler, but even auto-1 idle instance and 15s delay did nothing to solve the problem. Killing the instances just makes them come back again. Average QPS = 0.003, Average Latency = 64ms. Many of the instances are older than 15min.

There seems to be an issue recorded for this already: https://code.google.com/p/googleappengine/issues/detail?id=10893&q=Type%3DDefect&sort=-opened&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log%20Opened

Please star it if you have the same problem, to raise the prio.

Cheers, Andres

Answer 1

Yes there was an App Engine issue with instances yesterday. App Engine launched a lot of instances without remove them. Google has communicated on this Google group : https://groups.google.com/forum/#!topic/google-appengine-downtime-notify/VmH7s-Eeyso

The problem is now solved and extra instances will not be charged.