Something wrong with how I'm bundling rasterio into an executable

Question

Expected behavior and actual behavior.

I expected to compile a script using rasterio into an executable using pyinstaller. The script runs fine from within my python environment. However I am not able to freeze it into an executable using PyInstaller.

Steps to reproduce the problem.

I have a script called workflow_3.py which contains the following:

import rasterio

That's it. I tried to compile then run this using pyinstaller as follows:

(wps_env36) D:\11202750-002_RA2CE\Basis>pyinstaller workflow_3.py (wps_env36) D:\11202750-002_RA2CE\Basis>dist\workflow_3\workflow_3.exe

The compilation seems to run to completion, however when I run the executable I get the following error:

(wps_env36) D:\11202750-002_RA2CE\Basis>dist\workflow_3\workflow_3.exe
Traceback (most recent call last):
  File "workflow_3.py", line 1, in <module>
    import rasterio
  File "c:\programdata\anaconda2\envs\wps_env36\lib\site-packages\PyInstaller\loader\pyimod03_i
mporters.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\rasterio\__init__.py", line 23, in <module>
  File "rasterio\_base.pyx", line 1, in init rasterio._base
ModuleNotFoundError: No module named 'rasterio._shim'
[17536] Failed to execute script workflow_3

Attempt to fix the problem

I modified the spec file by explicitly adding 'rasterio._shim' to the list contained by the hidden-imports variable. Then I ran pyinstaller workflow_3.spec . This caused other ModuleNotFoundError for modules such as control.py, crs.py and vrt.py.

Adding these to hidden-imports successfully eliminates the ModuleNotFoundError for that particular package but it still looks for other packages, all of which are contained in C:\ProgramData\Anaconda2\envs\wps_env36\Lib\site-packages\rasterio. There are about 40 modules in this directory. It seems excessive to add every single filename in this directory to the hidden-imports variable. In fact I don't even know if it would work.

Therefore, I also tried adding that whole directory into my pathex variable so that it can extend the PYTHONPATH with it. However this causes another problem:

File "c:\programdata\anaconda2\envs\wps_env36\lib\traceback.py", line 5, in <module> File "c:\programdata\anaconda2\envs\wps_env36\lib\linecache.py", line 11, in <module> File "c:\programdata\anaconda2\envs\wps_env36\lib\tokenize.py", line 27, in <module> ImportError: cannot import name 'open' pre-safe-import-module hook failed, needs fixing.

Operating system

Windows 7

Rasterio version and provenance

The rasterio version is 1.0.8, from conda-forge The python version is 3.6.6

I have two versions of pyinstaller

pyinstaller               3.4              py36h7602738_0    conda-forge
PyInstaller               3.5.dev0+b13e6b30b           <pip>

The second one is the development version, which I had to get because of this problem

Question

How do I use PyInstaller to freeze an application which uses rasterio?

Answer 1

The current solution that I came up with is to force feed hidden-imports variable all modules contained within C:\ProgramData\Anaconda2\envs\wps_env36\Lib\sitepackages\rasterio using the glob package. In my spec file I added some python code to do this:

# -*- mode: python -*-

block_cipher = None
import glob, os
rasterio_imports_paths = glob.glob(r'C:\ProgramData\Anaconda2\envs\wps_env36\Lib\site-packages\rasterio\*.py')
rasterio_imports = ['rasterio._shim']

for item in rasterio_imports_paths:
    current_module_filename = os.path.split(item)[-1]
    current_module_filename = 'rasterio.'+current_module_filename.replace('.py', '')
    rasterio_imports.append(current_module_filename)

a = Analysis(['workflow_3.py'],
             pathex=['D:\\11202750-002_RA2CE\\Basis'],
             binaries=[],
             datas=[],
             hiddenimports=rasterio_imports,
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='workflow_3',
          debug=True,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='workflow_3')

Unfortunately this does not explain why pyinstaller was not able to see those modules in the first place. However it does momentarily solve this problem, and the code compiles fine.