Google cloud datastore slow (>800ms) with simple query from compute engine

Question

When I try to query the Google Cloud Datastore from a (micro) compute engine, it usually takes >800ms to get a reply. The best I got was 450ms, the worst was >3 seconds.

I was under the impression that latency should be much, much lower (like 20-80ms), so I'm guessing I'm doing something wrong.

This is the (node.js) code I'm using to query (from a simple datastore with just a single entity):

const Datastore = require('@google-cloud/datastore');
const projectId = '<my-project-id>';

const datastoreClient = Datastore({
  projectId: projectId
});

var query = datastoreClient.createQuery('Test').limit(1);

console.time('query');
query.run(function (err, test) {
  if (err) {
    console.log(err);
    return;
  }
  console.timeEnd('query');
});

Not sure if it's relevant, but my app-engine project is in the US-Central region, as is the compute engine I'm running the query from.

UPDATE

After some more testing I found out that the default authentication (token?) that you get when using the Node.js library provided by Google expires after about 4 minutes.
So in other words: if you use the same process, but you wait 4 minutes or more between requests, query times are back to >800ms.
I also tried authenticating using a keyfile, and that seemed to do better: subsequent request are still faster, but the initial request only takes half the time (>400ms).

Answer 1

This latency that you see for your initial requests to the Datastore are most likely due to caching being warmed up. The Datastore uses a distributed architecture to manage scaling, which allows your queries to scale with the size of your result set. By performing more of the same query, the better prepared the Datastore is to serve your query, and the more consistent the speeds of your results.

If you want similar result speeds on low Datastore access rates, it is recommended to configure your own caching layer. Google App Engine provides Memcache which is optimized for use with the Datastore. Since you are making requests from Compute Engine, you can use other third-party solutions such as Redis or Memcached.