fail to insert data into local Google App Engine datastore

Question

I am following the example from Google's Mobile Shopping Assistant sample which asks me to import data according to this link.

I tried the steps according to the example (all the sample codes are vanilla, i didn't change any thing except to fix the warning to use the latest Gradle build version)

I believe that I am missing an essential step that is not stated in the example. Can someone provide some insights into which steps I did wrongly?

the following are the steps i did

1. start local googleAppEngine app "backend"

2. ran cmd " appcfg.py upload_data --config_file bulkloader.yaml --url=http ://localhost:8080/remote_api --filename places.csv --kind=Place -e [email protected]".

   This command is supposed to insert 2 rows into the datastore (places.csv contains two entries)

3. this gave me the following readout

10:07 AM Uploading data records.

[INFO    ] Logging to bulkloader-log-20151020.100728

[INFO    ] Throttling transfers:

[INFO    ] Bandwidth: 250000 bytes/second

[INFO    ] HTTP connections: 8/second

[INFO    ] Entities inserted/fetched/modified: 20/second

[INFO    ] Batch Size: 10

Error 302: --- begin server output ---

--- end server output ---

4. I then go to "http://localhost:8080/admin/buildsearchindex" which displays "MaintenanceTasks completed."

5. Next I go to "http://localhost:8080/_ah/admin" but it displays

Datastore has no entities in the Empty namespace. You need to add data

programatically before you can use this tool to view and edit it.

Answer 1

I had the same problem but not with the local developer server but with the deployed version. After nearly going insane, I found a workaround to upload the data using appcfg. In my case, I noticed that when trying the following

Input not working for me:

gcloud auth login
appcfg.py upload_data --config_file bulkloader.yaml --url=http://<yourproject>.appspot.com/remote_api --filename places.csv --kind=Place --email=<[email protected]>

Output of error:

11:10 AM Uploading data records.
[INFO    ] Logging to bulkloader-log-20160108.111007
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Error 302: --- begin server output ---

--- end server output ---

as expected, I was not asked to authenticate myself again during the second command but apparently, appcfg could still not authenticate my account. I am using Win7, with Python 2.7, the Python Google App Engine SDK including appcfg.py and gcloud from the Google Cloud SDK if I get it right.

However, on https://cloud.google.com/container-registry/docs/auth it is shown that you can print out the access token using the gcloud command and then insert it manually in your appcfg command which worked for me

Input working for me:

gcloud auth login
gcloud auth print-access-token

This prints out the access token which you can use with appcfg

appcfg.py upload_data --oauth2_access_token=<oauth2_access_token> --config_file bulkloader.yaml --url=http://<yourproject>.appspot.com/remote_api --filename places.csv --kind=Place --email=<[email protected]>

Output of successful data upload:

10:42 AM Uploading data records.
[INFO    ] Logging to bulkloader-log-20160108.104246
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20160108.104246.sql3
[INFO    ] Connecting to <yourproject>.appspot.com/remote_api
[INFO    ] Starting import; maximum 10 entities per post
.
[INFO    ] 3 entities total, 0 previously transferred
[INFO    ] 3 entities (4099 bytes) transferred in 1.7 seconds
[INFO    ] All entities successfully transferred

I hope this helps anybody trying to solve this problem. To me, it is not clear what the source of this problem is, but this is a workaround that works for me.

BTW, I observed the same problem on a mac.

Answer 2

So here is what I found through testing. I went through the same steps initially and got the same error, but what is worthy of note in the output is the entry INFO client.py:669 access_token is expired:

MobileAssistant-Data> appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8080/remote_api --filename places.csv --kind=Place -e [email protected]
05:12 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20151112.171238
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
2015-11-12 17:12:38,466 INFO client.py:669 access_token is expired. Now: 2015-11-12 22:12:38.466000, token_expiry: 2015-11-06 01:33:21
Error 302: --- begin server output ---

This looked somewhat like an issue I saw in the Remote API handler for the dev server that surfaced after ClientLogin was deprecated (but in the Python SDK). Just to test I edited the build.gradle to use the latest SDK version (1.9.28 from 1.9.18) and ran it again:

MobileAssistant-Data> appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8080/remote_api --filename places.csv --kind=Place -e [email protected]
05:16 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20151112.171615
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
2015-11-12 17:16:15,177 INFO client.py:669 access_token is expired. Now: 2015-11-12 22:16:15.177000, token_expiry: 2015-11-06 01:33:21
2015-11-12 17:16:15,565 INFO client.py:669 access_token is expired. Now: 2015-11-12 22:16:15.565000, token_expiry: 2015-11-06 01:33:21
2015-11-12 17:16:15,573 INFO client.py:571 Refreshing due to a 401 (attempt 1/2)
2015-11-12 17:16:15,575 INFO client.py:797 Refreshing access_token
2015-11-12 17:16:16,039 INFO client.py:571 Refreshing due to a 401 (attempt 2/2)
2015-11-12 17:16:16,040 INFO client.py:797 Refreshing access_token
... (ad infinitum)

This output now mirrors the Python Remote API issue exactly. So it seems that the same issue exists with the Java Remote API (the auth check has not been properly updated to use the new auth scheme).

The workaround in Python was to manually edit the SDK source code and stub out the auth check. I suspect a similar hack would be necessary for the Java SDK. It's not as straightforward though as the SDK would need to be rebuilt from source.

If I come across anything else I will update this answer with my findings. Note that this should work perfectly fine with a deployed application - it's only the dev server that is affected.

Update:

The culprit is the admin check in com/google/apphosting/utils/remoteapi/RemoteApiServlet.java as with the same issue in the Python SDK noted above. Unfortunately you cannot trivially rebuild the SDK from source, as the build target in build.xml only includes 'jsr107cache' and the rest of the build is done from pre-generated binaries. It looks like we'll have to just wait until this is fixed in a future release, but for now I will update the public bug.

For now I would recommend sticking to the documentation and only using the deployed app version for remote_api uploads.