Copy configuration file on installation

Question

I am trying to package my Python project, which comes with a configuration dotfile that I want copied into the user's home directory on installation. The quick guide to packaging says that this can be done using the data_files argument to setuptools.setup. So this is what I have:

data_files = [(os.path.expanduser("~"), [".my_config"])]

This appears to work fine if I use python setup.py install, but when I upload my package to PyPI and run pip install the dotfile isn't copied.

FWIW, I've put the dotfile in the MANIFEST.in and also tried including the package_data argument to setup. None of these steps appear to make a difference. If I pip install and poke around the site-packages directory, just the source files are here.

How can I achieve what I'm looking for?

Answer 1

This is an issue I had once to experience myself. Its root is that when you are building a wheel file, all the absolute paths specified in data_files will be relativized to the target site-packages directory, see this issue on github. This influences installations performed by pip install as it will build a wheel out of any source package (.tar.gz, .tar.bz2 or .zip) and install the resulting wheel:

$ pip install spam-0.1.tar.gz 
Processing ./spam-0.1.tar.gz
Building wheels for collected packages: spam
  Running setup.py bdist_wheel for spam ... done
  Stored in directory: /Users/hoefling/Library/Caches/pip/wheels/d0/95/be/bc79f1d589d90d67139481a3e706bcc54578fdbf891aef75c0
Successfully built spam
Installing collected packages: spam
Successfully installed spam-0.1

Checking installed files yields:

$ pip show -f spam
Name: spam
Version: 0.1
...
Location: /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages
Requires: 
Files:
  Users/hoefling/.my_config
  spam-0.1.dist-info/DESCRIPTION.rst
  spam-0.1.dist-info/INSTALLER
  spam-0.1.dist-info/METADATA
  spam-0.1.dist-info/RECORD
  spam-0.1.dist-info/WHEEL
  spam-0.1.dist-info/metadata.json
  spam-0.1.dist-info/top_level.txt

Note the path meant to be absolute is relative to the Location dir. In the example, .my_config would be placed under /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages/Users/hoefling/.my_config.

It gets even better because these built wheels are cached on your disk, so next time you reinstall the package and the built wheel still exists in pip's cache, it will be used for the installation and you won't even see any mentions of building a wheel in the terminal log.

There is no real solution to avoid this. The most decent workaround I found is to prohibit "binary" packages when installing to enforce the execution of package's setup.py on installation:

$ pip install spam-0.1.tar.gz --no-binary=spam
Processing ./spam-0.1.tar.gz
Skipping bdist_wheel for spam, due to binaries being disabled for it.
Installing collected packages: spam
  Running setup.py install for spam ... done
Successfully installed spam-0.1

The file is now placed correctly:

$ pip show -f spam
Name: spam
Version: 0.1
...
Location: /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages
Requires: 
Files:
  ../../../../../.my_config
  spam-0.1-py3.6.egg-info/PKG-INFO
  spam-0.1-py3.6.egg-info/SOURCES.txt
  spam-0.1-py3.6.egg-info/dependency_links.txt
  spam-0.1-py3.6.egg-info/top_level.txt

Unfortunately, the user must be separately informed about calling pip install with the extra key (via readme, webpage FAQ or similar) as there is no possibility to prohibit building the wheel in package metadata.

As the result, I do not include files with absolute paths anymore. Instead, I install them with the python sources in the site-packages dir. In the python code, I have to add additional logic for the existence checks and file copying if necessary:

# program entrypoint

if __name__ == '__main__':
    config = os.path.join(os.path.expanduser('~'), '.my_config')
    if not os.path.exists(config):
        shutil.copyfile('.my_config', config)
    main.run()

Answer 2

Besides what @hoefling said, I suggest you not using data_files at all! Because it's really unpredictable where the files will be copied to. You could test this by giving the directory something like '', '/', or '/anything/you/want'.

I suggest you use package_data instead, which just copies the files under the distributed package root on installation. Then you can copy that to anywhere you want at run time.

For more on package_data, refer to Python Doc https://docs.python.org/2/distutils/setupscript.html#installing-package-data