Why 'python3 -m venv myenv' installs older version of pip into myenv than any version of pip I can find anywhere on the system?

Question

This is not causing me any problem that I can't solve by activating the virtual environment and running pip install -U pip, but I always wonder where the older version of pip is coming from.

I'm using OS X 10.7.5. When I create a virtual environment using either pyvenv-3.4 myenv or python3 -m venv myenv, the version of pip that is installed inside the virtual environment is 6.0.8, but I have upgraded my global pip to 6.1.1.

Here is a terminal session demonstrating what I mean:

$ python3 -m venv myenv
$ myenv/bin/pip -V
pip 6.0.8 from /Users/dust/Desktop/myenv/lib/python3.4/site-packages (python 3.4)

Here is what I would like to occur:

$ source myenv/bin/activate
(myenv)$ pip -V
UPDATED SYSTEM VERSION HERE WOULD BE NICE

I can't find a pip 6.0.8 anywhere else, other than what is created inside virtual environments.

Here are the outputs of various commands that I have use to try and figure this out:

$ which pip
/Library/Frameworks/Python.framework/Versions/3.4/bin/pip

$ which pip3
/Library/Frameworks/Python.framework/Versions/3.4/bin/pip3

$ pip -V
pip 6.1.1 from /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages (python 3.4)

$ pip3 -V
pip 6.1.1 from /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages (python 3.4)

I even tried using find:

$ find / -type f -name pip 2>&1 | awk '! /^f.*$/'
/Library/Frameworks/Python.framework/Versions/3.4/bin/pip
/usr/local/bin/pip

$ find / -type f -name pip3 2>&1 | awk '! /^f.*$/'
/Library/Frameworks/Python.framework/Versions/3.4/bin/pip3

I thought maybe that the /usr/local/bin/pip might have been the culprit, but no:

$ /usr/local/bin/pip -V
pip 6.1.1 from /Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg (python 2.7)

Hmm. Perhaps the OS X python has it?

$ /usr/bin/python
>>> import pip
>>> pip.__version__
'6.1.1'

6.1.1 is reported no matter which distribution of python I ask, whether it be OS X's 2.7.1, python.org's 2.7.9, or python.org's 3.4.3.

Is it possible (or advisable) to update the version of pip that gets put into a virtual environment?

Answer 1

I face the same issue, running OSX 10.10.2 and python 3.4.2. Most recently I created a virtual environment in a debian wheezy machine with python 3.4.3 and also ended up with an older version of pip than available. had to upgrade pip.

I've been upgrading pip within the virtual environment to 6.1.1 from 6.0.8 manually, because I'm o.c.d about software library versions that way - and yes, I am upgrading my python 3 version to 3.4.3 right now. Anyway, my system's python3-pip is the latest version 6.1.1, so I've also wondered why pyvenv creates a new virtual environment and loads it with old pip.

I haven't noticed anything bad happen in any of the virtual environments due to upgrading pip, (but on the flip side, I haven't noticed anything good either) Apparently the new pip is faster -- didn't notice, and outputs less junk on successful installs because user's don't care -- also didn't notice, probably because i'm one of those that don't care, and also comes with a state-of-the art coffee machine capable of latte art to boot!!! -- still waiting on sudo pip install latte to finish :(

So, to answer your question, it is definitely possible, and probably advisable to upgrade, because apparently the new pip fixes some bugs and goes faster, but I guess the speed up isn't that major, and the bug fixes don't affect all that many people (I've never faced a bug with my usage of the old pip).

You can link to system site-packages using the flag --system-site-packages when you create a new virtual environment, like this

pyvenv myenv --system-site-packages

This will link to your system wide version of pip, and would remove the annoyance that is manually upgrading pip on every virtual environment, but if you do this, then is your virtual environment all that virtual?

update: following my rant above, I went into the venv package's source to dig. pip is set up by a method called _setup_pip in the file __init__.py, line 248

def _setup_pip(self, context):
        """Installs or upgrades pip in a virtual environment"""
        # We run ensurepip in isolated mode to avoid side effects from
        # environment vars, the current directory and anything else
        # intended for the global Python environment
        cmd = [context.env_exe, '-Im', 'ensurepip', '--upgrade',
                                                    '--default-pip']
        subprocess.check_output(cmd, stderr=subprocess.STDOUT)

So, venv seems to be calling ensurepip from the shell using the subprocess module.

One more minute of google-fu gave me this from the documentation for ensurepip.

ensurepip.version()

Returns a string specifying the bundled version of pip that will be installed when bootstrapping an environment.

So, from the command line, the following code:

$ python3 -c 'import ensurepip; print(ensurepip.version())' 
6.0.8

displays my current pip that will be bootstrapped with ensurepip.

I guess we're stuck with the old version of pip for every new install until ensurepip gets upgraded, as I can't find a way to upgrade the version of pip that comes with ensurepip

Answer 2

Newer

If you want to "hotpatch" your installed python, just modify the versions listed in ensurepip/__init__.py and replace the two files in ensurepip/_bundled. You can find this location by running find * | grep ensurepip from the directory where python is installed. On macOS with Homebrew, this is the location: /usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ensurepip

You will also want to delete the ensurepip/__pycache__ directory that contains the .pyc files.

My older, build-time fix:

You are able to update the bundled versions of pip and setuptools by patching Python before building it from source. The following patch will update the bundled versions of pip and setuptools to the current available today. You will want to invoke configure with the following option: --with-ensurepip=upgrade

Those whl files are downloaded from PYPI here:

https://pypi.org/project/pip/#files

https://pypi.org/project/setuptools/#files

diff -ru Python-3.7.1/Lib/ensurepip/__init__.py Python-3.7.1.new/Lib/ensurepip/__init__.py
--- Python-3.7.1/Lib/ensurepip/__init__.py  2018-10-20 06:04:19.000000000 +0000
+++ Python-3.7.1.new/Lib/ensurepip/__init__.py  2018-11-27 02:36:19.301655008 +0000
@@ -8,9 +8,9 @@
 __all__ = ["version", "bootstrap"]


-_SETUPTOOLS_VERSION = "39.0.1"
+_SETUPTOOLS_VERSION = "40.6.2"

-_PIP_VERSION = "10.0.1"
+_PIP_VERSION = "18.1"

 _PROJECTS = [
     ("setuptools", _SETUPTOOLS_VERSION),
Only in Python-3.7.1/Lib/ensurepip/_bundled: pip-10.0.1-py2.py3-none-any.whl
Only in Python-3.7.1.new/Lib/ensurepip/_bundled: pip-18.1-py2.py3-none-any.whl
Only in Python-3.7.1/Lib/ensurepip/_bundled: setuptools-39.0.1-py2.py3-none-any.whl
Only in Python-3.7.1.new/Lib/ensurepip/_bundled: setuptools-40.6.2-py2.py3-none-any.whl