How to create local copy of GAE datastore?

Question

I want to make client version of GAE app that store exact data of online version.(myapp.appspot.com) If i can use sdk instead, is any library or tools to sync online and sdk version? I try using bulkloader but i can't load downloaded data to local SDK? Please help.

Answer 1

As explained in this article (link updated, thanks to Zied Hamdi)

You simply need to enable the remote api

builtins:
- remote_api: on

Update your application then run the following commands:

appcfg.py download_data -A s~YOUR_APP_NAME --url=http://YOUR_APP_NAME.appspot.com/_ah/remote_api/ --filename=data.csv
appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data .

Edit for After April 12th 2016 on Latest AppEngine SDK:

The above works for SDK version 1.9.0 and before. However with the depreciation of ClientLogin, the above will cause an error of

03:13 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20160909.151355
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20160909.151355.sql3
2016-09-09 15:13:55,175 INFO client.py:578 Refreshing due to a 401 (attempt 1/2)
2016-09-09 15:13:55,176 INFO client.py:804 Refreshing access_token
2016-09-09 15:13:55,312 INFO client.py:578 Refreshing due to a 401 (attempt 2/2)

Recommended by Anssi here, we can use the API server directly without running into this error. For a typical dev_appserver startup you get the following output

INFO     2016-09-09 19:27:11,662 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO     2016-09-09 19:27:11,899 api_server.py:205] Starting API server at: http://localhost:52497
INFO     2016-09-09 19:27:11,905 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO     2016-09-09 19:27:11,918 admin_server.py:116] Starting admin server at: http://localhost:8000

instead of the above for upload use the API port, in this case

appcfg.py --url=http://localhost:52497/_ah/remote_api/ --filename=data.csv upload_data .

Answer 2

See the docs for details on how to download and upload your entire datastore. Simply bulk download from production, then bulk upload to your local datastore.

Bear in mind, however, that the local datastore is not designed to handle large volumes of data - you may run into performance or memory issues.