Google App-Engine memcached extremely slow

Question

I recently launched my app for iPhone/Android running with AppEngine backend. This is my first experience using AppEngine in Production.

As I get more traffic, I am starting to experience serious latency issues. Currently minimum idle instance is 1, max_pending_latency is 1s.

Yes, there are rooms for optimizations on my side, however I do not understand

1. Why the latency is not correlated with request/sec, traffic, memoryUsage, memcacheUsage, anything. I do not understand why there was no significant latency on Sep 21.

2. Why the call to memcached needs to be as slow as 500ms. (Usually it is 10 times faster). I am using NDB and 1GB dedicated memcached. Increasing to 5GB had no effect.

Is this simply how AppEngine works? I would like to get your insight.

Thanks

Answer 1

I forgot to update this.... I remember the issue was caused by me creating datastore keys by myself. Basically, not well distributed keys introduced "hot tablet" problem. Once I stopped creating my own keys and let the AppEngine create them, the issue seemed to be resolved.