Scrapy throws ImportError: cannot import name xmlrpc_client

Question

After install Scrapy via pip, and having Python 2.7.10:

scrapy Traceback (most recent call last): File "/usr/local/bin/scrapy", line 7, in <module> from scrapy.cmdline import execute File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 48,   in <module> from scrapy.spiders import Spider File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py",     line 10, in <module> from scrapy.http import Request File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line    12, in <module> from scrapy.http.request.rpc import XmlRpcRequest File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py",   line 7, in <module> from six.moves import xmlrpc_client as xmlrpclib ImportError: cannot import name xmlrpc_client 

But I can import module:

Python 2.7.10 (default, Jun 10 2015, 19:42:47)  [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import scrapy >>>  

What's going on?

Answer 1

I've just fixed this issue on my OS X.

Please backup your files first.

sudo rm -rf /Library/Python/2.7/site-packages/six* sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six* sudo pip install six 

Scrapy 1.0.0 is ready to go.

---

If you encounter an error rm: /System/Library/... Operation not permitted

Please try to disable System Integrity Protection
See Operation Not Permitted when on root El capitan (rootless disabled)

Answer 2

This is a known issue on Mac OSX for Scrapy. You can refer to this link.

Basically the issue is with the PYTHONPATH in your system. To solve the issue change the current PYTHONPATH to point to the newer or none Mac OSX version of Python. Before running Scrapy, try:

export PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH

If that worked you can change the .bashrc file permanently:

echo "export PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH" >> ~/.bashrc

If none of this works, take a look at the link above.