Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to install/compile pip requirements in parallel (make -j equivalent)

I have a lot of packages to install in my pip requirement and I'd like to process them in parallell.

I know that, for example, that if I want n parallel jobs from make I have to write make -j n; is there an equivalent command for pip requirements?

Thanks!

like image 840
Rowandish Avatar asked Oct 31 '14 07:10

Rowandish


People also ask

How do I install pip to another version?

How do I Install a Specific Version of a Python Package? To install a specific version of a Python package you can use pip: pip install YourPackage==YourVersion . For example, if you want to install an older version of Pandas you can do as follows: pip install pandas==1.1. 3 .

Is pip Cache thread safe?

As this discussion is the first hit on google: pip should not be considered thread safe. Plus there is a bug in pip that might cause an exception when invoking pip from within a thread if you are not running pip.

Does pip install requirements in order?

To be clear, pip makes no promises to install the packages in the order they are declared in the requirements file. The only promise is "dependencies before dependant".

Does pip install for both Python 2 and 3?

In this article, learn how to install pip on Ubuntu 18.04. Note: If you are using Python in a virtual environment created with pyvenv or virtualenv, then pip is available regardless of the version of Python in use. This also applies to Python 2.7. 9 or newer (Python series 2) and Python 3.4 or later (Python series 3).


1 Answers

Sometimes pip uses make to build dependencies. If before it starts you set MAKEFLAGS like:

export MAKEFLAGS="-j$(nproc)"
pip install -r requirements.txt

This may help building native dependencies.

Note: nproc resovles as the number of CPUs in your system.

like image 58
Eddy del Valle Avatar answered Oct 03 '22 19:10

Eddy del Valle