How to recover from pip freeze exception?

Question

I am installing python 2.7 packages on an ubuntu 14 system and have run into a problem wherein "pip freeze" produces output like this ...

$ pip freeze
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run
    req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist
    assert len(specs) == 1 and specs[0][0] == '=='
AssertionError

Storing debug log for failure in /home/ubuntu/.pip/pip.log


$ cat /home/ubuntu/.pip/pip.log
------------------------------------------------------------
/usr/bin/pip run on Tue Feb  9 21:00:36 2016
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run
    req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist
    assert len(specs) == 1 and specs[0][0] == '=='
AssertionError

I am uncertain what caused this state of affairs.
My suspicion is that a failure of the easy_install program might have caused something to get corrupted.

I have tried re-installing pip.
This had no effect.

I suspect that I will have to do a complete uninstall and reinstall of all python packages via pip and python related ubuntu packages via apt-get. package by package.

This will be quite time consuming and I am uncertain if it will achieve the desired result.

Is there a better approach to this problem?

Here are the packages that I installed prior to running easy_install:

sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev protobuf-compiler gfortran libjpeg62 libfreeimage-dev libatlas-base-dev git python-dev python-pip libgoogle-glog-dev libbz2-dev libxml2-dev libxslt-dev libffi-dev libssl-dev libgflags-dev liblmdb-dev python-yaml python-numpy

Here is the initial exception that I got from easy_install:

$ sudo easy_install pillow
Searching for pillow
Reading https://pypi.python.org/simple/pillow/
Best match: Pillow 3.1.1
Downloading https://pypi.python.org/packages/source/P/Pillow/Pillow-3.1.1.zip#md5=3868f54fd164e65f95fbcb32f62940ae
Processing Pillow-3.1.1.zip
Writing /tmp/easy_install-Bvu_2f/Pillow-3.1.1/setup.cfg
Running Pillow-3.1.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Bvu_2f/Pillow-3.1.1/egg-dist-tmp-lMm5Tk
warning: no previously-included files found matching '.editorconfig'
Building using 4 processes
_imaging.c: In function ‘getink’:
_imaging.c:474:9: warning: ‘r’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     int r, g, b, a;
         ^
libImaging/Resample.c:87:45: warning: always_inline function might not be inlinable [-Wattributes]
 static float __attribute__((always_inline)) i2f(int v) {
                                             ^
Building using 4 processes
Building using 4 processes
Building using 4 processes
--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 3.1.1
platform     linux2 2.7.6 (default, Jun 22 2015, 17:58:13)
             [GCC 4.8.2]
--------------------------------------------------------------------
*** TKINTER support not available
--- JPEG support available
*** OPENJPEG (JPEG2000) support not available
--- ZLIB (PNG/ZIP) support available
--- LIBTIFF support available
--- FREETYPE2 support available
*** LITTLECMS2 support not available
*** WEBP support not available
*** WEBPMUX support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.

To check the build, run the selftest.py script.

Adding Pillow 3.1.1 to easy-install.pth file
Installing pildriver.py script to /usr/local/bin
Installing viewer.py script to /usr/local/bin
Installing gifmaker.py script to /usr/local/bin
Installing pilconvert.py script to /usr/local/bin
Installing pilfont.py script to /usr/local/bin
Installing pilfile.py script to /usr/local/bin
Installing createfontdatachunk.py script to /usr/local/bin
Installing explode.py script to /usr/local/bin
Installing pilprint.py script to /usr/local/bin
Installing player.py script to /usr/local/bin
Installing thresholder.py script to /usr/local/bin
Installing painter.py script to /usr/local/bin
Installing enhancer.py script to /usr/local/bin

Installed /usr/local/lib/python2.7/dist-packages/Pillow-3.1.1-py2.7-linux-x86_64.egg
Processing dependencies for pillow
Finished processing dependencies for pillow
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 330, in _handle_workers
    debug('worker handler exiting')
TypeError: 'NoneType' object is not callable

Exception Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 357, in _handle_tasks
    debug('task handler got sentinel')
TypeError: 'NoneType' object is not callable

TypeError: TypeError("'NoneType' object does not support item deletion",) in <Finalize object, dead> ignored
ubuntu@ip-10-234-31-217:~/nvidia_installers/cuda$ 

I was subsequently able to install pillow via pip.

I've noticed that there seem to be 2 different dist-packages in my path.
That seems to be a little odd.

>>> import sys
>>> import pprint
>>> pp = pprint.PrettyPrinter(indent=4)
>>> pp.pprint(sys.path)
[   '',
    '/usr/lib/python2.7',
    '/usr/lib/python2.7/plat-x86_64-linux-gnu',
    '/usr/lib/python2.7/lib-tk',
    '/usr/lib/python2.7/lib-old',
    '/usr/lib/python2.7/lib-dynload',
    '/usr/local/lib/python2.7/dist-packages',
    '/usr/lib/python2.7/dist-packages']

Here is the full procedure that I am following:
Install Caffe on EC2 from scratch

I understand that the version of pip supported by apt-get is very old.
I have tried updating pip.
I get "Owned by OS" errors.
I am uncertain about how to diverge from using apt-get here and the consequences of doing so.

$ pip install -U pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-8.0.2-py2.py3-none-any.whl#md5=2056f553d5b593d3a970296f229c1b79
  Downloading pip-8.0.2-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, owned by OS
  Can't roll back pip; was not uninstalled
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1436, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 672, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 902, in move_wheel_files
    pycompile=self.pycompile,
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 206, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 193, in clobber
    os.makedirs(destsubdir)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip-8.0.2.dist-info'

Storing debug log for failure in /home/ubuntu/.pip/pip.log

$ sudo pip install -U pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-8.0.2-py2.py3-none-any.whl#md5=2056f553d5b593d3a970296f229c1b79
  Downloading pip-8.0.2-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, owned by OS
Successfully installed pip
Cleaning up...

$ pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)
ubuntu@ip-10-234-31-217:~$ pip freeze
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run
    req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist
    assert len(specs) == 1 and specs[0][0] == '=='
AssertionError

Storing debug log for failure in /home/ubuntu/.pip/pip.log

Another method of upgrading pip also failed:

$ sudo pip install --upgrade pip
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Requirement already up-to-date: pip in /usr/local/lib/python2.7/dist-packages
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

$ pip freeze
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run
    req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist
    assert len(specs) == 1 and specs[0][0] == '=='
AssertionError

Storing debug log for failure in /home/ubuntu/.pip/pip.log

$ ls -lF /home/ubuntu/.cache/pip/http
ls: cannot access /home/ubuntu/.cache/pip/http: No such file or directory
$ ls -lF /home/ubuntu/.cache
total 0
-rw-r--r-- 1 ubuntu ubuntu 0 Feb  6 00:39 motd.legal-displayed

Answer 1

To install pip on debian based linux distros (including ubuntu), I've had good results using easy_install, from the python-setuptools package. So that typically looks like:

sudo apt-get install python-setuptools
sudo easy_install pip

It looks like you already have easy_install available. Anyway, before trying to install pip that way, you should try to clean up as much of your current setup as you can. I would remove the python-pip package, remove any other traces of upgraded pip, and also remove traces of the pillow module, since that is what you were installing when things started going wrong. Maybe something like:

sudo apt-get remove --purge python-pip
sudo rm -rf /usr/local/lib/python2.7/{dist,site}-packages/pip*
sudo rm -rf /usr/local/lib/python2.7/{dist,site}-packages/pillow*

Finally, when you use easy_install or pip to install something which includes a command in addition to an import-able python module, the command will be put in /usr/local/bin, so you should make sure that /usr/local/bin is in your shell's PATH. The pip command will be in there. (I'd only use easy_install to install pip, then use pip to install any other python package.)

EDIT: by the way, the "initial exception from easy_install pillow" is not an exception, nor an error. It shows some non-fatal compiler warnings, and that some features were not enabled, but it appears to have worked. It was probably later actions with pip which caused a problem for pip.

much later edit 2: I had a "braino" and wrote /usr/local/lib where I meant /usr/local/bin in the last original paragraph above. Fixed now.