pyInstaller changing dll and pyd output location

Question

I'm trying to use pyInstaller to package a wxpython application. I looking for a variation of the "one-folder" mode whereby the dlls and pyds are not stored in the top-level directory but in a subdirectory instead (like "dlls" or "libs").

This is the spec file currently:

# -*- mode: python -*-
import os

a = Analysis\
(
    ["..\\job_scraper\\load_gui.py"],
    pathex        = ["C:\\Users\\Administrator\\Documents\\Projects\\python\\PyInstaller\\load_gui"],
    hiddenimports = [],
    hookspath     = None,
    runtime_hooks = None
)

a_binaries = []
for (name, path, data_type) in a.binaries:
    (non_ext, ext) = os.path.splitext(name)
    if(ext in [".pyd", ".dll"]):
        a_binaries.append((os.path.join("libs", name), path, data_type))
    else:
        a_binaries.append((name, path, data_type))

a.binaries = TOC(a_binaries)

pyz = PYZ(a.pure)

exe = EXE\
(
    pyz,
    a.scripts,
    exclude_binaries = True,
    name             = "load_gui.exe",
    debug            = False,
    strip            = None,
    upx              = True,
    console          = False
)

coll = COLLECT\
(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    [("control.csv", "..\\job_scraper\\control.csv", "DATA")],
    strip = None,
    upx   = True,
    name  = "load_gui"
)

This does to put the dlls (not the pyds) into a lib folder, however it seems to do this after linking and so the program fails to launch because it can't find the expected dlls.

Answer 1

The problem is that the sys.path doesn't include your subdirectories. So when the program runs, it doesn't know where to look for your .dll or .pyd files.

Of course putting the code sys.path.append("relative/path/to/your/subdirectories") on top of your main script would come to mind. But one again, this code is only executed after everything is loaded and in place.

According to this blog, the solution is using the runtime hook of pyinstaller. Runtime hook tells the bootstrapping code to run any of your arbitrary code before your main script runs - before importing any stuff.

1. Preparation

Create a hooker.py which will add all your custom paths to sys.path. Put it somewhere and do it 1 time only.

import sys
import os
sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), "lib"))    # for pyd
sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), "windll")) # for dll

2. Tell pyinstaller to include the hooker.py

with spec file:

a = Analysis\
(
    ["..\\job_scraper\\load_gui.py"],
    pathex        = ["C:\\Users\\Administrator\\Documents\\Projects\\python\\PyInstaller\\load_gui"],
    hiddenimports = [],
    hookspath     = None,
    runtime_hooks = "absolute/path/to/hooker.py"  # <----- add it here
)

or with commandline:

pyinstaller --runtime-hook="absolute/path/to/hooker.py" the_rest_parameters

3. Run pyinstaller

As usually, it will create dist/your_main_script_name folder which contains the exe file, manifest, library.zip, and a bunch of .dll and .pyd

4. Create custom folders

Now you can create a windll folder and lib or anything that you add to sys.path at step 1. Then move all .pyd files to lib and all .dll files to windll.

Run your exe and it will crash! So move back these below files to parent folder.

• pythonXX.dll , where XX is your python version
• VCRUNTIME140.dll
• pywintypesXX.dll, where XX is your python version and if it is included

These files are needed for the bootstrap so we cannot move them without modifying the bootstrap code.

Run the exe again and it should work normally.

---

Soon you will get bored of repeating all of the above over and over again. So here is what I have been doing.

Create compiler.bat with the content similar to:

pyinstaller --runtime-hook="absolute/path/to/hooker.py" --onedir --icon path/to/icon ^
--exclude-module=UnNeeded_module_A ^
--exclude-module=UnNeeded_module_B ^
%1

@echo off
for %%F in (%1) do set pyi_output=%%~nxF
set pyi_output=%pyi_output:~0,-3%

mkdir dist\%pyi_output%\windll
mkdir dist\%pyi_output%\lib

move dist\%pyi_output%\*.dll dist\%pyi_output%\windll
move dist\%pyi_output%\*.pyd dist\%pyi_output%\lib

move dist\%pyi_output%\windll\python36.dll dist\%pyi_output%
move dist\%pyi_output%\windll\VCRUNTIME140.dll dist\%pyi_output%
if exist dist\%pyi_output%\windll\pywintypes36.dll (
    move dist\%pyi_output%\windll\pywintypes36.dll dist\%pyi_output%
)

pause

To not mess up your project, create a copy of your code folder and place this compile.bat inside. Then, just drag and drop your main_script.py to compile.bat.

The pause command keeps the console windows open so that you know if the compilation is successful or not.