Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Cheapest way to import over 10m entities to app engine

Is there a cheap way to import bulk data into app engine without paying for the datastore write operations?

I estimate that loading my 10m+ entities would cost 100+ dollars to get my data into app engine is there any other way? It seems like (accounting for indexs) each entity is taking 10 write operations can I cut it down to 1 write operation somehow?

Thanks

like image 278
mike Avatar asked Nov 13 '22 13:11

mike


1 Answers

App Engine maintains built-in indexes for each indexed property (i.e. those that don't have indexed=False as a parameter when defining the property). See http://code.google.com/appengine/articles/index_building.html for more details.

If there are properties that won't appear in any of your queries, specify indexed=False as you are defining the property. This will save you the cost of the index write operations associated with such properties that you don't need to be automatically indexed.

like image 56
cv12 Avatar answered Feb 12 '23 14:02

cv12