Export Google Cloud Datastore and import to BigQuery programmatically

Question

I'm looking for a method to export my Cloud Datastore and import it into BigQuery daily. The manual way is described at google page. I do not find a clean way to automate it.

Answer 1

There isn't a simple way to do this, but you can separate out the two parts: creating appengine backups and loading them into bigquery.

You can use scheduled backups to create datastore backups periodically (https://cloud.google.com/appengine/articles/scheduled_backups).

You can then use Apps Script to automate the BigQuery portion (https://developers.google.com/apps-script/advanced/bigquery#load_csv_data) or use an AppEngine cron to do the same thing.