Data Migrations and AppEngine

Question

I've done a lot of development in rails, and am looking into developing projects using python & app engine.

From the demo project and what I've seen so far, I've got a question/concern about app engine projects:

How is data migration handled in app-engine? For example, if I change the name of an entity/table (ex: Texts to Documents), or change a column in an existing table (ex: age to dob) - how does it handle old data when this happens?

Thanks!

Answer 1

The short answer is: It doesn't handle it. You can't change the name of an entity, you can change a property but you'll have to update the data manually.

Your Model definitions are just your applications "view" of how to interpret the entities stored in the datastore. If I had a definition like:

class MyEntity(db.Model):
    text = db.TextProperty()

And run my application for a while filling up the text property of my enties, Then later renamed the column to:

class MyEntity(db.Model):
    description = db.TextProperty()

All my existing data would stay exactly as it was (lots of entities in the datastore with populated text properties. Only when I tried to load the entities into my model instances I would only see them as empty entities (with no description set, and no way to access the text data that currently exists). Saving (Putting) my entity back into the datastore would then overwrite the old data, and the data would be lost.

If you make changes to your schema like this, or more likely just changing a field type. It will be up to you to pre-process your data to handle the changes. The model-layer will raise errors if you try and load an entity that no-longer conforms to your model definitions.

To help with this manual task of updating your data the weapons of choice are:

1. remote_api / remote_api_shell.py
2. The mapreduce library (especially the "mapper" part)

With the remote_api[1] setup you can open an interactive Python session to your live data, and run scripts locally (mostly) as if they are running directly on the production servers. I find this is the fastest easiest way to fix/cleanup data for smallish one-off tasks.

The mapper api[2] could be employed if you have a much larger task, say altering millions of entities and want to take advantage of doing as much of this in parallel as possible.