On OS X El Capitan I can not upgrade a python package dependent on the six compatibility utilities NOR can I remove six

Question

I am trying to use scrape, but I have a problem.

from six.moves import xmlrpc_client as xmlrpclib

ImportError: cannot import name xmlrpc_client

Then, I tried pip install --upgrade six scrape, but:

  Found existing installation: six 1.4.1     DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project. Uninstalling six-1.4.1: Exception: Traceback (most recent call last):   File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 211, in main status = self.run(options, args)   File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 311, in run root=options.root_path,   File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 640, in install requirement.uninstall(auto_confirm=True)   File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 716, in uninstall paths_to_remove.remove(auto_confirm)   File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 125, in remove renames(path, new_path)   File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 315, in renames shutil.move(old, new)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move copy2(src, real_dst)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2 copystat(src, dst)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat os.chflags(dst, st.st_flags)  OSError: [Errno 1] Operation not permitted: '/var/folders/3h/r_2cxlvd1sjgzfgs4xckc__c0000gn/T/pip-5h86J8-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info' 

Answer 1

Quick Fix:

I just got around what I think was the same problem. You might consider trying this (sudo, if necessary):

pip install scrape --upgrade --ignore-installed six

Github is ultimately where I got this answer (and there are a few more suggestions you may consider if this one doesn't solve your problem). It also seems as though this is an El Capitan problem.

Also, this technically might be a duplicate. But the answer the other post came up with was installing your own Python rather than relying on the default osx Python, which strikes me as more laborious.

---

Longer, Real Fix:

Eventually I ran into a situation where I actually needed to upgrade six in order to install some other libraries, at which point ignoring wasn't enough. The easiest way to upgrade six for me was to not use pip, but instead to manually download the .tar.gz from the six page, decompress, move to the newly decompressed six package directory, and run the installation manually (sudo, if necessary):

python setup.py install

Thanks to this answer for the guidance.

Answer 2

I don't think this is a duplicate, but actually this issue discussed here on the pip GitHub repository issues list.

NOTE: Depending on which package you're installing, you may need to modify the command referenced in the solution. In my case below, I was trying to setup virtualenv and virtualwrapper. If you're failing on installing aws-cli or any other package, just insert that into the script

The core of the problem is tied to Apple's new SIP that they shipped with El Capitan. More specifically,

OS X 10.11's python retains its own copy of six which is unremoveable, because of modifications Apple has done to their python distribution. 1.4.1 is not the latest, 1.10.0 is. It also comes early on their python's import path, so it will typically override later versions you install.

I would suggest using a different python for now. Python.org's, or installed via Homebrew, or Anaconda Python.

There is an incredibly detailed discussion on the Ask Different Stack Exchange that covers how the problems with SIP have been identified, addressed, and evolved since the original release of El Capitan. Although I found it fascinating, you'll spend less time following the instructions below than it would take you to read it, so I'd reccomend checking it out AFTER you finish the following...

I ran into the exact same error when attempting to upgrade VirtualEnv & VirtualEnvWrapper. There were several suggestions kicked around on that above thread, but in the end the most stable was to

1. Leverage the built-in support for the sudo OPTION to specify a HOME environment variable

$ man sudo    -H   The -H (HOME) option option sets the HOME environment variable          to the home directory of the target user (root by default) as specified          HOME environment variable depends on sudoers(5) settings.  By default,          sudo will set HOME if env_reset or always_set_home are set, or if         set_home is set and the -s option is specified on the command line. 

2. Leverage pip's options to force an upgrade and ignore any pre-existing packages

$ pip install --help | grep upgrade    -U, --upgrade   Upgrade all specified packages to the newest available          version. This process is recursive regardless of whether a dependency          is already satisfied.  beejhuff@ignatius:~/mac_setup$ pip install --help | grep ignore-installed    -I, --ignore-installed  Ignore the installed packages (reinstalling instead). 

First, my original attempt & error:

$ sudo pip install virtualenv virtualenvwrapper  The directory '/Users/beejhuff/Library/Caches/pip/http' or its parent directory  is not owned by the current user and the cache has been disabled.  Please check the permissions and owner of that directory. If executing  pip with sudo, you may want sudo's -H flag.  The directory '/Users/beejhuff/Library/Caches/pip' or its parent directory  is not owned by the current user and caching wheels has been disabled.  check the permissions and owner of that directory. If executing pip with  sudo, you may want sudo's -H flag.  Collecting virtualenv   Downloading virtualenv-15.0.0-py2.py3-none-any.whl (1.8MB)     100% |████████████████████████████████| 1.8MB 335kB/s  Collecting virtualenvwrapper   Downloading virtualenvwrapper-4.7.1-py2.py3-none-any.whl Collecting virtualenv-clone (from virtualenvwrapper)   Downloading virtualenv-clone-0.2.6.tar.gz Collecting stevedore (from virtualenvwrapper)   Downloading stevedore-1.12.0-py2.py3-none-any.whl Collecting pbr>=1.6 (from stevedore->virtualenvwrapper)   Downloading pbr-1.8.1-py2.py3-none-any.whl (89kB)     100% |████████████████████████████████| 92kB 362kB/s  Collecting six>=1.9.0 (from stevedore->virtualenvwrapper)   Downloading six-1.10.0-py2.py3-none-any.whl Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper   Running setup.py install for virtualenv-clone ... done   Found existing installation: six 1.4.1     DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.     Uninstalling six-1.4.1: Exception: Traceback (most recent call last):   File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/basecommand.py", line 209, in main     status = self.run(options, args)   File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/commands/install.py", line 317, in run     prefix=options.prefix_path,   File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_set.py", line 726, in install     requirement.uninstall(auto_confirm=True)   File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_install.py", line 746, in uninstall     paths_to_remove.remove(auto_confirm)   File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove     renames(path, new_path)   File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/utils/__init__.py", line 267, in renames     shutil.move(old, new)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move     copy2(src, real_dst)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2     copystat(src, dst)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat     os.chflags(dst, st.st_flags) OSError: [Errno 1] Operation not permitted: '/tmp/pip-GQL8Gi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info' 

The Solution

It required modifying my installation command in THREE specific ways:

1. I had to add the -H flag to sudo
2. I had to add the --upgrade option AFTER the name of the package I was upgrading (virtualenv)
3. I had to use the --ignore-installed flag and specify the six package was the one to be ignored.
4. *Note: general command is $ sudo -H pip install <packagename> --upgrade --ignore-installed six - replace<packagename> with the specific package you need to install`

Final Working Example

1st Upgrade virtualenv

$ sudo -H pip install virtualenv --upgrade --ignore-installed six  Password:  Collecting virtualenv   Using cached virtualenv-15.0.0-py2.py3-none-any.whl Collecting six   Using cached six-1.10.0-py2.py3-none-any.whl Installing collected packages: virtualenv, six Successfully installed six-1.4.1 virtualenv-15.0.0 

2nd Upgrade virtualenvwrapper

$ sudo -H pip install virtualenvwrapper --upgrade --ignore-installed six  Password:    Downloading virtualenvwrapper-4.7.1-py2.py3-none-any.whl Collecting six   Downloading six-1.10.0-py2.py3-none-any.whl Collecting virtualenv (from virtualenvwrapper)   Downloading virtualenv-15.0.0-py2.py3-none-any.whl (1.8MB)     100% |████████████████████████████████| 1.8MB 751kB/s  Collecting virtualenv-clone (from virtualenvwrapper)   Downloading virtualenv-clone-0.2.6.tar.gz Collecting stevedore (from virtualenvwrapper)   Downloading stevedore-1.12.0-py2.py3-none-any.whl Collecting pbr>=1.6 (from stevedore->virtualenvwrapper)   Downloading pbr-1.8.1-py2.py3-none-any.whl (89kB)     100% |████████████████████████████████| 92kB 417kB/s  Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper   Running setup.py install for virtualenv-clone ... done Successfully installed pbr-1.8.1 six-1.4.1 stevedore-1.12.0 virtualenv-15.0.0 virtualenv-clone-0.2.6 virtualenvwrapper-4.7.1