Mac OS X El Capitan - Scrapy/Python ImportError: cannot import name xmlrpc_client

Question

I am trying to use Scrapy on Mac OS X El Capitan. I have zsh installed and I have tried everything that I could find online to fix this issue. I have also looked at Scrapy throws ImportError: cannot import name xmlrpc_client and could not resolve my problem!

Python installed via brew and added "pip install scrapy":

➜  DriverEBV  which python
/usr/local/bin/python

My .zshrc has the following line:

export PATH=/usr/local/bin:$PATH
export PYTHONPATH="/Library/Python/2.7/site-packages"

This is the error I get:

➜  DriverEBV  scrapy runspider DriverEBV.py
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line 12, in <module>
    from scrapy.http.request.rpc import XmlRpcRequest
  File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py", line 7, in <module>
    from six.moves import xmlrpc_client as xmlrpclib
ImportError: cannot import name xmlrpc_client

When I run "pip install scrapy" now this is what I see:

➜  DriverEBV  pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in /Library/Python/2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.8.0 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): service-identity in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=0.7 in /usr/local/lib/python2.7/site-packages (from pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): characteristic>=14.0.0 in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): enum34 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.1.0 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/site-packages (from cffi>=1.1.0->cryptography>=0.7->pyOpenSSL->scrape)

Anybody able to help me?

Answer 1

I've had nothing but pain mucking with the Mac OS X system Python libraries installed in the /Library/Python directory. What has worked well for me is a combination of MacPorts and virtualenv:

1. Install MacPorts

2. Install Python, pip, and virtualenv from MacPorts:

   /opt/local/bin/port install python27
   /opt/local/bin/port install py27-pip
   /opt/local/bin/port install py27-virtualenv
   

3. Setup virtualenv:

   /opt/local/bin/virtualenv-2.7 myenv
   

4. Activate virtualenv (don't forget the dot!)

   . myenv/bin/activate
   

5. Install scrapy

   pip install scrapy
   

This way, the system Python libraries are untouched and you can install whatever packages you like without having to remove or upgrade existing packages.

Answer 2

What helped me was to uninstall six and scrapy and then install again:

pip uninstall six
pip uninstall scrapy

pip install six
pip install scrapy

Run with sudo if necessary.

---

Or, you can also try upgrading six and scrapy:

pip install --upgrade scrapy
pip install --upgrade six