GAE NDB "Result cannot be set twice" error

Question

I hit this Result cannot be set twice Runtime error on GAE Ndb async queries when doing ndb.Future.wait_all(futures) on a bunch of async queries.

Something like this:

futures = []
for item in items:
    item._future_get = MyEntity.query(...).get_async()
    futures.append(item._future_get)

ndb.Future.wait_all(futures)
# ...

It fails on the wait_all with Result cannot be set twice

This error message is nowhere mentionned on SO. Google has 2-3 mentions of it dating back to 2011, and with no clear explanation.

More info:

items are ndb entities from a previous fetch. But they don't really matter here (at least I think), since the query is performed on MyEntity. I am used to attaching futures to the object they relate to in this way, so it's easier to sort out when all have completed.

The stack trace:

  File "/home/my_project/app/main/admin/my_module.py", line 166, in admin_base_cleanup_details ndb.Future.wait_all(futures)
  File "/usr/lib/python2.7/google_appengine/google/appengine/ext/ndb/tasklets.py", line 350, in wait_all ev.run1()
  File "/usr/lib/python2.7/google_appengine/google/appengine/ext/ndb/eventloop.py", line 235, in run1 delay = self.run0()
  File "/usr/lib/python2.7/google_appengine/google/appengine/ext/ndb/eventloop.py", line 197, in run0 callback(*args, **kwds)
INFO     2016-04-26 08:40:04,152 module.py:808] default: "GET /admin/cleanup/details?mode=status HTTP/1.1" 500 -
  File "/usr/lib/python2.7/google_appengine/google/appengine/ext/ndb/tasklets.py", line 475, in _on_future_completion self._help_tasklet_along(ns, ds_conn, gen, val)
  File "/usr/lib/python2.7/google_appengine/google/appengine/ext/ndb/tasklets.py", line 386, in _help_tasklet_along self.set_result(result)
  File "/usr/lib/python2.7/google_appengine/google/appengine/ext/ndb/tasklets.py", line 265, in set_result
    raise RuntimeError('Result cannot be set twice.')
RuntimeError: Result cannot be set twice.

Some more precisions:

• Yes, it does happen on GAE as well as on local dev.

• No, it does not fail every single time, but often enough.

I found it has to do with concurrency from another thread. The web page started 2 requests through ajax calls: one for an update query with some async calls, that would take quite a few seconds, and another one like a periodic status update, quicker but also with async calls. It is the latter that failed, not always but very often. Since then, I avoided overlapping the two requests and it stopped failing. It still seems like a bug, since overlapping requests is not something forbidden.

Answer 1

You are using get_async() which "Asynchronously returns the first query result", whereas you should probably use fetch_async() to get a Future.

https://cloud.google.com/appengine/docs/python/ndb/queryclass#Query_get_async