ImportError cannot import name BytesIO when import caffe on ubuntu

Question

I am trying to make caffe running on my machine equipped with Ubuntu 12.04LTS. After finishing all the steps on the Installation page, I trained the LeNet model successfully and tried to use it as the tutorial from here. Then I got the following error:

Traceback (most recent call last): 
    File "<string>", line 1, in <module>
ImportError: No module named caffe
Error in sys.excepthook:
Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
      from apport.fileutils import likely_packaged, get_recent_crashes
    File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 1, in <module>
      from apport.report import Report
    File "/usr/lib/python2.7/dist-packages/apport/report.py", line 18, in <module>
      import problem_report
    File "/usr/lib/python2.7/dist-packages/problem_report.py", line 14, in <module>
      import zlib, base64, time, sys, gzip, struct, os
    File "/usr/lib/python2.7/gzip.py", line 10, in <module>
      import io
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO

Original exception was:
Traceback (most recent call last): 
    File "<string>", line 1, in <module>
ImportError: No module named caffe

I set the PYTHONPATH in .bashrc file before I did the above. What is the problem? Could anyone give some hint? I am really confused. After running the command python -c 'import io; print io.__file__'in the very directory:

Traceback (most recent call last): 
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO
Error in sys.excepthook:
Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
      from apport.fileutils import likely_packaged, get_recent_crashes
    File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 1, in <module>
      from apport.report import Report
    File "/usr/lib/python2.7/dist-packages/apport/report.py", line 18, in <module>
      import problem_report
    File "/usr/lib/python2.7/dist-packages/problem_report.py", line 14, in <module>
      import zlib, base64, time, sys, gzip, struct, os
    File "/usr/lib/python2.7/gzip.py", line 10, in <module>
      import io
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO

Original exception was:
Traceback (most recent call last): 
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO

So, the problem becomes to: how to solve the name issue? P.S.: I also inserted an issue at the repository of caffe.

Answer 1

You appear to have a package or module named io in your Python path that is masking the standard library package. It is imported instead but doesn't have a BytesIO object to import.

Try running:

python -c 'import io; print io.__file__'

in the same location you are running the tutorial and rename or move the file named by that import, presuming it is not the standard library version (ending in lib/python2.7/io.pyc).

It could be you set your Python path to the wrong directory. You should include path/to/caffe/python, not path/to/caffe/python/caffe, nor should you try and run python with the latter as your current working directory. In both cases caffe/python/caffe/io.py instead of the standard library version.

The installation instructions are not at fault here; they clearly tell you to use:

export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH

Note the lack of /caffe at the end of that path.

Answer 2

I ran into this problem as well, installing caffe on an AWS ubuntu 14.04 instance following the script as outlined on the BVLC github repo here: "Caffe on EC2 Ubuntu 14.04".

I have setup the python path as instructed. As diagnosed by @Martijn Pieters, the problem is that caffe is importing its own io library, which is then importing scikit-image's io library, which in turn is trying (but failing) to load the standard python io library (where BytesIO is located). Instead, due to the python path, when scikit-image tries to import BytesIO from the module io, it is circularly leading back to caffe's io module.

I also found that even when not trying to import caffe, but due to having set my python path to include caffe, that this same problem hits me elsewhere.

There are probably several ways to address this. But the essence is that the top-level import of caffe is at fault. To verify this, I altered the caffe code as follows:

1. I renamed .../caffe/io.py module to .../caffe/caffe_io.py to be safe (although with correct namespace care, this shouldn't be necessary)

2. I modified the import at the top of the pycaffe.py module from: import caffe.io to import caffe.caffe_io

3. I modified the import in __init__.py the same way (from import caffe.io to import caffe.caffe_io)

Now, when you import io from python, it won't pick up the io library in caffe. When you import caffe, it will import its custom caffe_io library, and all should be well. You may want to do a more thorough scan through the python caffe modules to ensure I haven't overlooked other places where the import needs to change.

I hope this helps. Perhaps when I have time, I'll issue a pull request with these (or similar) changes to the caffe github repo.