How to properly handle conflicting distutils libraries with pip?

Question

After upgrade of pip version to 10.0.0, installation with pip fails when there is a version conflict with a distutils installed package:

Cannot uninstall '***'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

This could be for PyYAML, pyOpenSSL, urllib3, chardet and so on.

I try to manage this issue by uninstalling corresponding packages such as;

python-yaml python-openssl python-urllib3 python-chardet

with apt-get (Ubuntu) and then install those libraries again with pip

However as it might be expected removal by apt-get also results removal of many dependent additional system packages which seems not a good practice:

The following packages will be REMOVED:
apt-xapian-index cloud-init landscape-client-ui-install oneconf python-aptdaemon python-aptdaemon.gtk3widgets python-chardet python-cupshelpers python-debian python-openssl python-pip python-requests python-ubuntu-sso-client python-urllib3 python-yaml sessioninstaller software-center ssh-import-id system-config-printer-common system-config-printer-gnome system-config-printer-udev ubuntu-desktop ubuntu-release-upgrader-gtk ubuntu-sso-client ubuntu-sso-client-qt update-manager update-notifier update-notifier-common

I also don't want to downgrade pip to an older version.

So what is the best practice to handle conflicting distutils libraries with pip?

Ps: I supposed pip is for easy management of Python libraries but this incident makes it enough complicated.

Answer 1

Virtual environment of Python might help to handle conflicting libraries even with newer versions of pip

Setup virtualenv

Python3 has builtin virtual environment. In case of Python2, virtualenv can be used for this purpose.

Use following commands to setup virtualenv

sudo pip install virtualenv
venv_path="${HOME}/py_venv"
mkdir -p "${venv_path}"
virtualenv "${venv_path}"

It can be activated by source command

source "${venv_path}/bin/activate"
(py_venv) my_user@my_machine:~$

and can be deactivated by deactivate command

(py_venv) my_user@my_machine:~$ deactivate
my_user@my_machine:~$

Confirm paths for python and pip

(py_venv) my_user@my_machine:~$ which python
/home/my_user/py_venv/bin/python
(py_venv) my_user@my_machine:~$ which pip
/home/my_user/py_venv/bin/pip

Be aware by default executing with sudo does not point to virtualenv

(py_venv) my_user@my_machine:~$ sudo which python
/usr/bin/python
(py_venv) my_user@my_machine:~$ sudo which pip
/usr/local/bin/pip

Install / uninstall packages with pip

(py_venv) my_user@my_machine:~$ pip install ansible

Successfully installed in virtualenv

(py_venv) my_user@my_machine:~$ which ansible
/home/my_user/py_venv/bin/ansible

Uninstall a conflicting system package in virtualenv

(py_venv) my_user@my_machine:~$ pip uninstall urllib3
Skipping urllib3 as it is not installed.

Uninstall same package in real environment

(py_venv) my_user@my_machine:~$ deactivate
my_user@my_machine:~$ pip uninstall urllib3
Cannot uninstall 'urllib3'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

As it can be seen with the help of Python virtual environment, it becomes possible to use newer versions of pip to install & uninstall Python libraries without touching any system packages.