How do I access my AppEngine DataStore entities from my Compute Engine VM?

Question

My app is running on App Engine, but I would like to access its NDB DataStore entities from my Compute Engine VM to do some processing and write the results back to the App Engine DataStore. How can I do that?

Also, are the Google Cloud DataStore and App Engine DataStore the same thing? https://developers.google.com/datastore/ https://developers.google.com/appengine/docs/python/ndb/

Answer 1

David's solution requires you to use App Engine instance time to make requests, but you can bypass it and make requests directly to Datastore from Compute Engine instance. There is a pretty good tutorial about how to do this. But its not so pretty like ndb.

>>> import googledatastore as datastore
>>> datastore.set_options(dataset='project-id')
>>> req = datastore.BeginTransactionRequest()
>>> datastore.begin_transaction(req)
<datastore.datastore_v1_pb2.BeginTransactionResponse object at ...>

Answer 2

Google Cloud Datastore is kind of a standalone version of App Engine's datastore.

Back to your problem, you have two options :

1. Write a web-service to expose your entities from the App Engine app to the Compute Engine VMs. One option is Cloud Endpoints. Cloud Endpoints uses OAuth2 authentication and the Compute Engine VMs are shipped with OAuth2 service accounts that you can use to authenticate to the service.

2. Use the App Engine remote API to access the Datastore. The remote API provides access to the Datastore as if you were in an App Engine instance. But by default the API requires you to provide the password of an App Engine amdin, so you might have to store your password in the Compute Engine VMs, which is not safe.