ModuleNotFoundError: No module named 'numpy.random._pickle'

Question

I have a doc2vec model which drives my recommendation app. I have built the doc2vec model and saved into s3 bucket. Now when i open the webapp the model should be loaded back from s3 but this not happenning. I used AWS Elasticbean stalk to deploy my webapplication. So, i then checked the logs of Elasticbean stalk for errors. This is the error it shows me when i try to deploy

-------------------------------------
/var/log/httpd/error_log
-------------------------------------
[Fri May 22 12:53:57.768095 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:53:57.768098 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     klass = self.find_class(module, name)
[Fri May 22 12:53:57.768104 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:53:57.768107 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     __import__(module, level=0)
[Fri May 22 12:53:57.768127 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:53:57.768138 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] 
[Fri May 22 12:53:57.768141 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] During handling of the above exception, another exception occurred:
[Fri May 22 12:53:57.768144 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] 
[Fri May 22 12:53:57.768150 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] Traceback (most recent call last):
[Fri May 22 12:53:57.768177 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/current/app/application.py", line 81, in <module>
[Fri May 22 12:53:57.768181 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     model_d2v = d2v.load_d2v('doc2vec_model', ENV)
[Fri May 22 12:53:57.768187 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/current/app/microservices/d2v_services.py", line 21, in load_d2v
[Fri May 22 12:53:57.768190 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     model=joblib.load(model_name)
[Fri May 22 12:53:57.768196 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 605, in load
[Fri May 22 12:53:57.768199 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     obj = _unpickle(fobj, filename, mmap_mode)
[Fri May 22 12:53:57.768204 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
[Fri May 22 12:53:57.768208 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     obj = unpickler.load()
[Fri May 22 12:53:57.768213 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1050, in load
[Fri May 22 12:53:57.768216 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     dispatch[key[0]](self)
[Fri May 22 12:53:57.768222 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:53:57.768230 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     klass = self.find_class(module, name)
[Fri May 22 12:53:57.768237 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:53:57.768240 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     __import__(module, level=0)
[Fri May 22 12:53:57.768250 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:54:07.192783 2020] [:error] [pid 8177] loading...doc2vec_model
[Fri May 22 12:54:12.237091 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] mod_wsgi (pid=8177): Target WSGI script '/opt/python/current/app/application.py' cannot be loaded as Python module.
[Fri May 22 12:54:12.237334 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] mod_wsgi (pid=8177): Exception occurred processing WSGI script '/opt/python/current/app/application.py'.
[Fri May 22 12:54:12.238604 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] Traceback (most recent call last):
[Fri May 22 12:54:12.238658 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/current/app/microservices/d2v_services.py", line 14, in load_d2v
[Fri May 22 12:54:12.238664 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     model=joblib.load(model_name)
[Fri May 22 12:54:12.238675 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 605, in load
[Fri May 22 12:54:12.238679 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = _unpickle(fobj, filename, mmap_mode)
[Fri May 22 12:54:12.238686 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
[Fri May 22 12:54:12.238690 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = unpickler.load()
[Fri May 22 12:54:12.238695 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1050, in load
[Fri May 22 12:54:12.238699 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     dispatch[key[0]](self)
[Fri May 22 12:54:12.238705 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:54:12.238709 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     klass = self.find_class(module, name)
[Fri May 22 12:54:12.238714 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:54:12.238718 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     __import__(module, level=0)
[Fri May 22 12:54:12.238737 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:54:12.238793 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] 
[Fri May 22 12:54:12.238800 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] During handling of the above exception, another exception occurred:
[Fri May 22 12:54:12.238803 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] 
[Fri May 22 12:54:12.238810 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] Traceback (most recent call last):
[Fri May 22 12:54:12.238840 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/current/app/application.py", line 81, in <module>
[Fri May 22 12:54:12.238845 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     model_d2v = d2v.load_d2v('doc2vec_model', ENV)
[Fri May 22 12:54:12.238851 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/current/app/microservices/d2v_services.py", line 21, in load_d2v
[Fri May 22 12:54:12.238855 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     model=joblib.load(model_name)
[Fri May 22 12:54:12.238872 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 605, in load
[Fri May 22 12:54:12.238883 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = _unpickle(fobj, filename, mmap_mode)
[Fri May 22 12:54:12.238889 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
[Fri May 22 12:54:12.238892 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = unpickler.load()
[Fri May 22 12:54:12.238897 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1050, in load
[Fri May 22 12:54:12.238901 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     dispatch[key[0]](self)
[Fri May 22 12:54:12.238906 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:54:12.238909 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     klass = self.find_class(module, name)
[Fri May 22 12:54:12.238914 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:54:12.238917 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     __import__(module, level=0)
[Fri May 22 12:54:12.238928 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:56:22.918445 2020] [mpm_prefork:notice] [pid 8172] AH00169: caught SIGTERM, shutting down 

I tried importing 'numpy.random._pickle' but it didn't work.

The code I used for loading the model:

import subprocess
from sklearn.externals import joblib
import gensim

model = load_d2v("doc2vec_model")

def load_d2v(fname):
    model_name = fname
    s3_base_path='s3://sd-flikku/datalake/current_doc2vec_model'
    path = s3_base_path+'/'+model_name  
    command = "aws s3 cp {} {}".format(path,model_name).split()
    print('loading...'+model_name)
    subprocess.call(command)
    model=joblib.load(model_name)
    return model

Can you please help me with this? Thanks in advance.

Answer 1

Upgrade the numpy library using:

pip install numpy --upgrade

Occurs when the model was built with a specific version of the library and tested with other versions.