How to install data_files to absolute path?

Question

I use pip with setuptools to install a package. I want pip to copy some resource files to, say, /etc/my_package.

My setup.py looks like this:

setup(
    ...
    data_files=[('/etc/my_package', ['config.yml'])]
)

When running pip install, the file ends up in

~/.local/lib/python3.5/site-packages/etc/my_package/config.yml

instead of /etc/my_package.

What am I doing wrong?

(pip version 9.0.1)

Answer 1

Short answer: use pip install --no-binary :all: to install your package.

I struggled with this for a while and eventually figured out that there is some weirdness/inconsistency in how data_files are handled between binary wheels and source distributions. Specifically, there is a bug with wheels that makes all paths in data_files relative to the install location (see https://github.com/pypa/wheel/issues/92 for an issue tracking this).

"Thats fine", you might say, "but I'm not using a wheel!". Not so fast! It turns out recent versions of pip (I am working with 9.0.1) will try to compile a wheel even from a source distribution. For example, if you have a package my_package you can see this doing something like

$ python setup.py sdist   # create source tarball as dist/my_package.tar.gz
              [...]
$ pip install dist/my_package.tar.gz # install the generated source
              [...]
Building wheels for collected packages: my_package
  Running setup.py bdist_wheel for my_package ... done

pip tries to be helpful and build a wheel to install from and cache for later. This means you will run into the above bug even though in theory you are not using bdist_wheel yourself. You can get around this by running python setup.py install directly from the package source folder. This avoids the building and caching of built wheels that pip will try to do but is majorly inconvenient when the package you want is already on PyPI somewhere. Fortunately pip offers an option to explicitly disable binaries.

$ pip install --no-binary :all: my_package
              [...]
Skipping bdist_wheel for my_package, due to binaries being disabled for it.
Installing collected packages: my_package
  Running setup.py install for my_package ... done
Successfully installed my_package-0.1.0

Using the --no-binary option prevents wheel building and lets us reference absolute paths in our data_files paths again. For the case where you are installing a lot of packages together and want to selectively disable wheels you can replace :all: with a comma separated list of packages.