Datastore: DatastoreFailureException: Unable to fetch global config [closed]

Question

Has anyone ever had this error before? I can't find a single shred of evidence on google that this has ever happened to anyone.

Here is the stacktrace which starts from a .delete() call on the datastore.

com.google.appengine.api.datastore.DatastoreFailureException: Unable to fetch global config
    at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:71)
    at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:129)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
    at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$7.get(AsyncDatastoreServiceImpl.java:406)
    at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl$7.get(AsyncDatastoreServiceImpl.java:402)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
    at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:76)
    at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:36)
    at com.google.appengine.api.datastore.DatastoreServiceImpl.delete(DatastoreServiceImpl.java:76)
    at com.universeprojects.cacheddatastore.CachedDatastoreService.delete(CachedDatastoreService.java:929)

Answer 1

We recently identified an issue with a bad instance in our infrastructure which caused timeouts for a limited number of configuration requests from AppEngine applications. The issue was resolved when the faulty instance was restarted ~6:00 AM Pacific Time 9/20/2016.

To prevent these errors in the future we are taking the following actions:

1. Modifying the retry behavior for configuration requests to better handle individual bad instances.
2. Implementing stricter monitoring policies around these instances to better detect these errors.