How do you resolve 'hidden imports not found!' warnings in pyinstaller for scipy?

Question

I'm working on using pyinstaller to create an .exe for a python program that uses pandas and sklearn. The pyinstaller process completes and produces the dist folder with the executable as expected. However, when I run the .exe I get module import errors related to sklearn and scipy.

I created a test script (test.py) to test imports, which only imports pandas and sklearn and then prints a success message:

import time
import pandas as pd
import sklearn

def main():
  print('hello world!')
  time.sleep(5)


if __name__ == '__main__':
  main()

I'm aware of pyinstaller hooks and I was able to resolve the pandas errors by adding a hook to the pyinstaller hooks directory. I added similar hooks for sklearn and scipy it looks like they're running, but in the pyinstaller output I'm getting warnings that 'Hidden import "sklearn.utils.sparsetools._graph_validation" not found!' and similar one for '._graph_tools'.

Here's the hook for scipy (hook-scipy.py):

print('loading custome hook for scipy')

from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('scipy') 

Here's a snapshot of the warnings generated from running pyinstaller

Here's a snapshot of the error when running test.exe

I'm working in a virtual environment where pyinstaller, pandas, sklearn, scipy and all dependencies are installed (at least I can get the regular test.py script running in this venv). Using PyInstaller 3.3.1, Python 3.6.4 on Windows 10.10.0.

Any help is appreciated!

Answer 1

You need to go into the hook-scipy.py (or create one) and have it look like this:

from PyInstaller.utils.hooks import collect_submodules
from PyInstaller.utils.hooks import collect_data_files
hiddenimports = collect_submodules('scipy')

datas = collect_data_files('scipy')

then go into the hook-sklearn.metrics.cluster.py file and modify it to look like this:

from PyInstaller.utils.hooks import collect_data_files

hiddenimports = ['sklearn.utils.sparsetools._graph_validation',
                 'sklearn.utils.sparsetools._graph_tools',
                 'sklearn.utils.lgamma',
                 'sklearn.utils.weight_vector']

datas = collect_data_files('sklearn')

I do not know if this part is necessary but I also created a hook-sklearn.py file that looks like this:

from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('sklearn')

In the cmd I used pyinstaller test.py -F to create one file.

Then it should work: