PyInstaller, how to include data files from an external package that was installed by pip?

Question

Problem

I am attempting to use PyInstaller to create an application for internal use within my company. The script works great from a working python environment, but loses something when translated to a package.

I know how to include and reference data files that I myself need within my package, but I am having trouble including or referencing files that should come in when imported.

I am using a pip-installable package called tk-tools, which includes some nice images for panel-like displays (looks like LEDs). The problem is that when I create a pyinstaller script, any time that one of those images is referenced, I get an error:

DEBUG:aspen_comm.display:COM23 19200 INFO:aspen_comm.display:adding pump 1 to the pump list: [1] DEBUG:aspen_comm.display:updating interrogation list: [1] Exception in Tkinter callback Traceback (most recent call last):   File "tkinter\__init__.py", line 1550, in __call__   File "aspen_comm\display.py", line 206, in add   File "aspen_comm\display.py", line 121, in add   File "aspen_comm\display.py", line 271, in __init__   File "aspen_comm\display.py", line 311, in __init__   File "lib\site-packages\tk_tools\visual.py", line 277, in __init__   File "lib\site-packages\tk_tools\visual.py", line 289, in to_grey   File "lib\site-packages\tk_tools\visual.py", line 284, in _load_new   File "tkinter\__init__.py", line 3394, in __init__   File "tkinter\__init__.py", line 3350, in __init__ _tkinter.TclError: couldn't open "C:\_code\tools\python\aspen_comm\dist\aspen_comm\tk_tools\img/led-grey.png": no such file or directory 

I looked within that directory in the last line - which is where my distribution is located - and found that there is no tk_tools directory present.

Question

How to I get pyinstaller to collect the data files of imported packages?

Spec File

Currently, my datas is blank. Spec file, created with pyinstaller -n aspen_comm aspen_comm/__main__.py:

# -*- mode: python -*-  block_cipher = None   a = Analysis(['aspen_comm\\__main__.py'],              pathex=['C:\\_code\\tools\\python\\aspen_comm'],              binaries=[],              datas=[],              hiddenimports=[],              hookspath=[],              runtime_hooks=[],              excludes=[],              win_no_prefer_redirects=False,              win_private_assemblies=False,              cipher=block_cipher)  pyz = PYZ(a.pure, a.zipped_data,              cipher=block_cipher)  exe = EXE(pyz,           a.scripts,           exclude_binaries=True,           name='aspen_comm',           debug=False,           strip=False,           upx=True,           console=True )  coll = COLLECT(exe,                a.binaries,                a.zipfiles,                a.datas,                strip=False,                upx=True,                name='aspen_comm') 

When I look within /build/aspen_comm/out00-Analysis.toc and /build/aspen_comm/out00-PYZ.toc, I find an entry that looks like it found the tk_tools package. Additionally, there are features of the tk_tools package that work perfectly before getting to the point of finding data files, so I know that it is getting imported somewhere, I just don't know where. When I do searches for tk_tools, I can find no reference to it within the file structure.

I have also tried the --hidden-imports option with the same results.

Partial Solution

If I 'manually' add the path to the spec file using datas = [('C:\\_virtualenv\\aspen\\Lib\\site-packages\\tk_tools\\img\\', 'tk_tools\\img\\')] and datas=datas in the Analysis, then all works as expected. This will work, but I would rather PyInstaller find the package data since it is clearly installed. I will keep looking for a solution, but - for the moment - I will probably use this non-ideal workaround.

If you have control of the package...

Then you can use stringify on the subpackage, but this only works if it is your own package.

Answer 1

I solved this by taking advantage of the fact that the spec file is Python code that gets executed. You can get the root of the package dynamically during the PyInstaller build phase and use that value in the datas list. In my case I have something like this in my .spec file:

import os import importlib  package_imports = [['package_name', ['file0', 'file1']]  datas = [] for package, files in package_imports:     proot = os.path.dirname(importlib.import_module(package).__file__)     datas.extend((os.path.join(proot, f), package) for f in files) 

And use the the resulting datas list as a parameters to Analysis.

Answer 2

Here's a one-liner using the same idea as Turn mentioned. In my case I needed a package (zbarcam) that was inside of kivy_garden. But I tried to generalize the process here.

from os.path import join, dirname, abspath, split from os import sep import glob import <package>  pkg_dir = split(<package>.__file__)[0] pkg_data = [] pkg_data.extend((file, dirname(file).split("site-packages")[1]) for file in glob.iglob(join(pkg_dir,"**{}*".format(sep)), recursive=True))