Sorting sys.path: first virtualenv, then /usr

Question

Why does sys.path contain /usr/... before directories from my virtualenv?

I create the virtualenv with --system-site-packages

The sys.path looks like this at the moment:

/home/my-virtualenv/src/foo
/usr/lib/python2.7/site-packages        <--- /usr paths should be below 
/usr/lib64/python2.7/site-packages
/home/my-virtualenv/lib/python27.zip
/home/my-virtualenv/lib64/python2.7
/home/my-virtualenv/lib64/python2.7/plat-linux2
/home/my-virtualenv/lib64/python2.7/lib-tk
/home/my-virtualenv/lib64/python2.7/lib-old
/home/my-virtualenv/lib64/python2.7/lib-dynload
/usr/lib64/python2.7
/usr/lib/python2.7
/usr/lib64/python2.7/lib-tk
/home/my-virtualenv/lib/python2.7/site-packages

I want all paths outside my virtualenv (/usr...) to be below the paths of the virtualenv.

Otherwise crazy things happen: I install a package with pip. Pip tells me that the new version is installed (pip freeze | grep -i ...) but the import does use the one from /usr/lib/python2.7/site-packages

I can't use --no-site-packages in my context.

Is there a way to sort sys.path?

Why I use system-site-packages

There seems to be no straight forward way to make single libraries from the global site-packages available in the virtualenv. See this question: make some modules from global site-packages available in virtualenv

There are packages like python-gtk which are very hard to install in a virtualenv.

Answer 1

Edit, after discussion:

"I want all paths outside my virtualenv (/usr...) to be below the paths of the virtualenv. [...] It needs to be sorted."

Then, just sort your sys.path before the first import happens. Given a certain path prefix corresponding to the location of your virtalenv, this approach likely is sufficient:

sys.path = sorted(sys.path, key=lambda x: x.startswith(prefix), reverse=True)

sorted()'s sorting behavior is stable: the original order is preserved for items with same sort key. Here, only two sort keys are used: True and False. You need to come up with a reliable way to set your prefix (You might want to hard-code it, or determine it based on the current working directory, I am sure you find a way).

Original answer (still valid, in general):

You did not want to elaborate on your requirements and application scenario too much, so I provide a more general answer: you may need to overthink your approach, and maybe not expect virtualenv to entirely solve your problem.

In certain situations, virtualenv is just not the perfect solution. It is a compromise, and one side of this compromise is exhaustively described in this article: https://pythonrants.wordpress.com/2013/12/06/why-i-hate-virtualenv-and-pip/

In many scenarios virtualenv serves a splendid purpose and does its job very fine! It helped me a lot, for sure, especially for development purposes. In other scenarios, it is either not a complete solution or even a bad solution.

So, there are a few different options now that I see:

1. Use virtualenv, but take control over those things that you need to have changed. For instance, modify sys.path from within your package before doing a certain import. While some might consider this "not clean", it for sure is a very quick way to efficiently and reliably control directory search order. sys.path.insert(1, foo) is actually used by many packages and test environments. It not uncommon at all. This approach may provide you with a working solution after just one minute of work. Give it a try!
2. If you think that virtualenv in your case does not behave as documented or clearly shows buggy behavior, then please report you findings to the project. They will surely appreciate your concise feedback.
3. If you think that virtualenv does not provide the level of isolation or control that you need in your application case, you might want to look at other options, such as Docker or Vagrant, or fully-features virtual machines.