"Too much contention" when creating new entity in dataStore

Question

This morning my GAE application generated several error log: "too much contention on these datastore entities. please try again.". In my mind, this type of error only happens when multiple requests try modify the same entity or entities in the same entity group.

When I got this error, my code is inserting new entities. I'm confused. Does this mean there is a limitation of how fast we can create new entity?

My code of model definition and calling sequence is show below:

# model defnition
class ExternalAPIStats(ndb.Model):
    uid = ndb.StringProperty()
    api = ndb.StringProperty()
    start_at = ndb.DateTimeProperty(auto_now_add=True)
    end_at = ndb.DateTimeProperty()

# calling sequence
stats = ExternalAPIStats(userid=current_uid, api="eapi:hr:get_by_id", start_at=start_at, end_at=end_at)
stats.put()  # **too much contention** happen here

That's pretty mysterious to me. I was wondering how I shall deal with this problem. Please let me know if any suggestion.

Answer 1

Without seeing how the calls are made(you show the calling code but how often is it called, via loop or many pages calling the same put at the same time) but I believe the issue is better explained here. In particular

You will also see this problem if you create new entities at a high rate with a monotonically increasing indexed property like a timestamp, because these properties are the keys for rows in the index tables in Bigtable.

with the 'start_at' being the culprit. This article explains in more detail.

Possibly (though untested) try doing your puts in batches. Do you run queries on the 'start_at' field? If not removing its indexes will also fix the issue.

How is the puts called (ie what I was asking above in a loop, multiple pages calling)? With that it might be easier to narrow down the issue.

Answer 2

Here is everything you need to know about Datastore Contention and how to avoid it: https://developers.google.com/appengine/articles/scaling/contention?hl=en (Deleted)

UPDATE: You are reaching writes per second limit on the same entity group. Default it is 1 write per second. https://cloud.google.com/datastore/docs/concepts/limits

Source: https://stackoverflow.com/a/47800087/1034622