What exactly is the throughput restriction on an entity group in Google App Engine's datastore?

Question

The documentation describes a limitation on the throughput to an entity group in the datastore, but is vague on what exactly the limitation is. My confusion is in two parts:

1. What is being restricted?

Specifically, is it:

1. The number of writes?
2. Number of transactions that write to the datastore?
3. Number of transactions regardless of whether it reads or writes to the datastore?

2. What is the type of the restriction?

Specifically, is it:

1. An artificially enforced one-per-second hard rule?
2. An empirically observed max throughput, that may in practice be better based on factors like network load, etc.?

Answer 1

There's no throughput restriction per se, but to guarantee atomicity in transactions, updates must be serialized and applied sequentially and in order, so if you make enough of them things will start to fail/timeout. This is called datastore contention:

Datastore contention occurs when a single entity or entity group is updated too rapidly. The datastore will queue concurrent requests to wait their turn. Requests waiting in the queue past the timeout period will throw a concurrency exception. If you're expecting to update a single entity or write to an entity group more than several times per second, it's best to re-work your design early-on to avoid possible contention once your application is deployed.

To directly answer your question in simple terms, it's specifically the number of writes per entity group (5/ish per second), and it's just a rule of thumb, your milage may vary (greatly).

Some people have reported no contention at all, while others have problems to get more than 1 update per second. As you can imagine this depends on the complexity of the operation and the load of all the machines involved in execution.