gcloud ml-engine local predict RuntimeError: Bad magic number in .pyc file

Question

My objective is to make predictions on google cloud ml engine.

I installed gcloud sdk on linux ubuntu 16.04LT following Google instructions. I already have a machine learning trained model. I using python version anaconda python 3.5.

I run:

gcloud ml-engine local predict --model-dir={MY_MODEL_DIR} --json-instances={MY_INPUT_JSON_INSTANCE}

I received the message: ERROR:

(gcloud.ml-engine.local.predict) RuntimeError: Bad magic number in .pyc file

Below is all the stack trace:

DEBUG: (gcloud.ml-engine.local.predict) RuntimeError: Bad magic number in .pyc file
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 797, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 757, in Run
    resources = command_instance.Run(args)
  File "/usr/lib/google-cloud-sdk/lib/surface/ml_engine/local/predict.py", line 65, in Run
    args.text_instances)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/local_utils.py", line 89, in RunPredict
    raise LocalPredictRuntimeError(err)
LocalPredictRuntimeError: RuntimeError: Bad magic number in .pyc file
ERROR: (gcloud.ml-engine.local.predict) RuntimeError: Bad magic number in .pyc file
Evaluation ended**

Answer 1

actually it works with python3, you just need to delete the pyc files in google cloud folders, so the prediction call can compile them with python3.

to know the location of the pyc files , i did enable the flag --verbosity debug in the prediction call:

gcloud ml-engine local predict --model-dir=${MODEL_LOCATION} --json-instances=data/new-data2.json --verbosity debug

the trackback will give you info about the path of gcloud ml engine files, in my machine was:

/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/

go to that directory and delete the pyc files.

Answer 2

In fact I myself post this question to help people with the same problem, because I couldn't find an easy concise answer.

There are other solutions, in my opinion even better than mine, but this was what solved for me.

My solution was that google cloud sdk doesn't works with python 3, at least in my configuration. To solve:

1. install a anaconda virtual environment with python 2 (in my case 2.7.14)
2. activate the environment
3. execute the gcloud command again

If your export ml model and inputs are OK that will work.

Simple problem, but caused a lot of pain to me, just because I couldn't easily find this pre-requirement or I simply missed it.

I hope help somebody.

Answer 3

Find and delete all the pyc files in the google SDK. They were compiled with the wrong python environment. They will be recompiled automatically when next needed.

%%bash
find "/tools/google-cloud-sdk/lib/" -name '*.pyc' -delete