GAE Datastore backup

Question

Is it necessary to do backups of GAE's Datastore?
Does anyone have any experience, suggestions, tricks for doing so?

Answer 1

Backups are always necessary to protect against human error. Since App Engine encourages you to build mutiple revisions of your code that run against the same dataset, it's important to be able to go back.

A simple dump/restore tool is explained in the Bulkloader documentation.

Something else I've done in the past for major DB refactors is:

1. Change the entity name in your new code (e.g. User -> Customer or User2 if you have to)
2. When looking up an entity by key:
   1. Try the key and return if possible
   2. Try the key for the old db.Model class. If you find it, migrate the data, put(), and return the new entity
3. Use the entity as usual

(You may have to use a task queue to migrate all the data. If you always fetch the entities by key it's not necessary.)

Deploy a new version of your code so that both coexist server-side. When you activate the new version, it is like a point-in-time snapshot of the old entities. In an emergency, you could reactivate the old version and use the old data.

Answer 2

You can now use the managed export and import feature, which can be accessed through gcloud or the Datastore Admin API:

Exporting and Importing Entities

Scheduling an Export