PYTHONPATH order on Ubuntu 14.04

Question

I have two computers running Ubuntu 14.04 server (let's call them A and B). B was initially a 10.04 but it has received two upgrades to 12.04 and 14.04. I do not understand why the python path is different on the two computers.

As you can see on the two paths below, the pip installation path /usr/local/lib/python2.7/dist-packages comes before the apt python packages path /usr/lib/python2.7/dist-packages on Ubuntu A, but it comes after on Ubuntu B.

This leads to several problems if a python package is installed both via apt and pip. As you can see below, if both python-six apt package and six pip package are installed, they may be two different library versions.

The installation of packages system is not always my choice, but might be some dependencies of other packages that are installed.

This problem could probably be solved with a virtualenv, but for reasons I will not detail, I cannot use virtualenv here, and must install pip packages system-wide.

Ubuntu A

>>> import sys, six
>>> sys.path
['',
 '/usr/local/bin',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/local/lib/python2.7/dist-packages/IPython/extensions']
>>> six
<module 'six' from '/usr/local/lib/python2.7/dist-packages/six.pyc'>

Ubuntu B

>>> import sys, six
>>> sys.path
['',
 '/usr/local/bin',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/local/lib/python2.7/dist-packages/IPython/extensions']
>>> six
>>> <module 'six' from '/usr/lib/python2.7/dist-packages/six.pyc'>

For both machines $PATH is the same, and $PYTHONPATH is empty.

• Why are those PYTHONPATHS different?

• How can I fix the pythonpath order in "Ubuntu B" so it will load pip packages before the system ones, in a system-wide way? Is there a apt package I should reinstall or reconfigure so the PYTHONPATH would prioritize pip packages ?

Answer 1

As we cannot explore into your system, I am trying to analysis your first question by illustrating how sys.path is initialized. Available references are where-does-sys-path-starts and pyco-reverse-engineering(python2.6).

The sys.path comes from the following variables(in order):

1. $PYTHONPATH (highest priority)
2. sys.prefix-ed stdlib
3. sys.exec_prefix-ed stdlib
4. site-packages
5. *.pth in site-packages (lowest priority)

Now let's describe each of these variables:

1. $PYTHONPATH, this is just a system environment variable.
2. & 3. sys.prefix and sys.exec_prefix are determined before any python script is executed. It is actually coded in the source Module/getpath.c.

The logic is like this:

IF $PYTHONHOME IS set:
    RETURN sys.prefix AND sys.exec_prefix as $PYTHONHOME
ELSE:
    current_dir = directory of python executable;
    DO:
        current_dir = parent(current_dir)
        IF FILE 'lib/pythonX.Y/os.py' EXSITS:
            sys.prefix = current_dir
        IF FILE 'lib/pythonX.Y/lib-dynload' EXSITS:
            sys.exec_prefix = current_dir
        IF current_dir IS '/':
            BREAK
    WHILE(TRUE)
    IF sys.prefix IS NOT SET:
       sys.prefix = BUILD_PREFIX
    IF sys.exec_prefix IS NOT SET:
       sys.exec_prefix = BUILD_PREFIX

4. & 5. site-packages and *.pth are added by import of site.py. In this module you will find the docs:

   This will append site-specific paths to the module search path. On Unix (including Mac OSX), it starts with sys.prefix and sys.exec_prefix (if different) and appends lib/python/site-packages as well as lib/site-python. ... ...

   For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python/dist-packages, Debian addons install into /usr/{lib,share}/python/dist-packages. /usr/lib/python/site-packages is not used.

   A path configuration file is a file whose name has the form .pth; its contents are additional directories (one per line) to be added to sys.path. ... ...

And a code snippet for important function getsitepackages:

sitepackages.append(os.path.join(prefix, "local/lib",
                            "python" + sys.version[:3],
                            "dist-packages"))
sitepackages.append(os.path.join(prefix, "lib",
                            "python" + sys.version[:3],
                            "dist-packages"))

Now I try to fig out where may be this odd problem comes from:

1. $PYTHONPATH, impossible, because it is empty both A and B
2. sys.prefix and sys.exec_prefix, maybe, please check them and as well as $PYTHONHOME
3. site.py, maybe, check the file.

The sys.path output of B is quite odd, dist-package (site-package) goes before sys.exec_prefix (lib-dynload). Please try to investigate each step of sys.path initialization of machine B, you may find out something.

Very sorry that I cannot replicate your problem. By the way, about your question title, I think SYS.PATH is better than PYTHONPATH, which makes me misinterpretation as $PYTHONPATH at first glance.