All dependencies are not downloaded with "pip download"

Question

I'm trying to setup a local directory with packages that can be reused for installation on a machine without an internet connection but I'm having problems with some of the packages.

I first download the packages with

pip download -r requirements.txt -d my_packages --no-binary :all:

And then I try to install them with

pip install -r requirements.txt --no-index -f my_packages

One of the packages I'm having trouble installing is elasticsearch-dsl==6.1.0:

pip install -r requirements --no-index -f my_packages
Looking in links: my_packages
Collecting elasticsearch-dsl==6.1.0
Collecting six (from elasticsearch-dsl==6.1.0)
Collecting python-dateutil (from elasticsearch-dsl==6.1.0)
  Installing build dependencies ... error
  Complete output from command /Users/Oskar/.pyenv/versions/2.7.15/envs/no_internet/bin/python2.7 -m pip install --ignore-installed --no-user --prefix /private/var/folders/36/t0_t6_td2f560t2j0149vjmw0000gn/T/pip-build-env-moib0N --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links my_packages -- setuptools wheel:
  Looking in links: my_packages
  Collecting setuptools
    Could not find a version that satisfies the requirement setuptools (from versions: )
  No matching distribution found for setuptools

Sure, setuptools I can manually install but there is more packages than that which is required for all the other packages. django-guardian==1.4.9 is another example which requires pytest-runner which for some reason is not downloaded with pip download

Answer 1

Because you specified --no-binary :all:, pip only downloads sdists for the packages and does not download any wheels. However, pip still needs each package in wheel form when it comes time to install, and so pip install tries to build a wheel for each package from the sdist, and this step requires setuptools, wheel, and anything listed in the package's setup_requires. I don't know why your environment doesn't have setuptools, but any healthy Python environment should have both it and wheel installed when the environment is created.

Unfortunately for you, packages listed in setup_requires are installed by setuptools itself rather than by pip, so pip download has no way to capture these requirements. If you insist on using --no-binary :all:, then the only solution is to manually compile a list of packages that are missing when pip install runs, download them, and then either install the missing packages before the ones that depend on them or else configure setuptools to look for the downloads in my_packages. Fortunately, the list will likely be rather short (probably just pytest-runner and setuptools_scm).

Answer 2

You can use pip2pi to avoid reinventing the wheel, this is a very handy tool for creating your own pip repos. I maintain a partial pypi mirror for my team and had the best results with this project.

From its Github page: "pip2pi builds a PyPI-compatible package repository from pip requirements"

$ pip install pip2pi
$ mkdir packages
$ pip2tgz packages/ -r requirements.txt
$ dir2pi -n packages/ # <--- this builds/updates the index, i.e. the "simple/" directory

Now just serve the packages/simple/ folder through http and use it as your index.

$ pip install -i http://myserver.net/simple/ -r requirements.txt

To avoid the hassle of pip security check you can tune your pip.conf like this:

[global]
format=columns
disable_pip_version_check = 1
index-url=file:http://myserver.net/simple/

Now you can install from your own mirror with the regular pip install command.