GAE NDB restore fails with "Unexpected error contacting datastore"

Question

I'm used to performing regular backup and restore of my NDB datastore, often restoring to a different 'project', where I will be testing a new version of the application. Up to now, it has worked fine.

Now, Ndb restore operation fails in a systematic way, with the trace in the log:

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/_internal/mapreduce/handlers.py", line 526, in handle
    ctx.flush()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/_internal/mapreduce/context.py", line 455, in flush
    pool.flush()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/datastore_admin/utils.py", line 695, in flush
    datastore._GetConnection()._reserve_keys(self.keys)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 2170, in _reserve_keys
    self._async_reserve_keys(None, keys).get_result()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 921, in get_result
    results = self.__rpcs[0].get_result()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 2211, in __reserve_keys_hook
    self.check_rpc_success(rpc)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1371, in check_rpc_success
    raise _ToDatastoreError(err)
InternalError: Unexpected error contacting datastore (2016-11-19T18:01:37+00:00).

Any clue ? I may have missed something, but I would swear nothing has changed as compared to last week's situation where such restore worked ok.

Just in case there was something wrong with my latest backup, I tried to restore an old backup, which had been successfully restored a couple of times. Same InternalError on restore. Including in a brand new GAE project.

Having tried many (many) times now, I found the 'unexpected errors' always happen for the same entities. The whole backup/restore involves about 30 different kinds, of which 5 fail with said error. Among these, in some cases there are entities that are restored, but not all, in other cases no entities have been restored. It's like some specific entities could be responsible for the exception. But, once again, this also happens for old backups, which had worked in the past.

There are few questions related to Ndb backup and restore (and fewer answers still). Don't Ndb applications use backups ?

Answer 1

So the final word on this is not hidden at the bottom of a long list of comments, I'll post an answer.

It seems this was a bug that had appeared a few days/weeks back, which a few users like me have experienced when restoring to another project.

The Google project team did a great job, fixing the issue promptly. Many thanks to @Ed Davisson.