PyInstaller and Pandas

Question

I have a fairly simple Python module that I am trying to compile into a Windows .exe file. In my script I am using the wxPython and Pandas libraries. The PyInstaller .exe file that is generated only works/opens when the Pandas library is excluded from my module.

I am getting the same issue whether I use --onefile or --onedir in PyInstaller. I found online that the "new" version of PyInstaller (2.1) should have taken care of this bug. Does anyone have any ideas on what to do?

PyInstaller: version 2.1
pandas: version 0.15.2
Python: version 2.7

Answer 1

I ran into the same issue. I boiled it down to a simple script like this Hello.py:

import pandas
print "hello world, pandas was imported successfully!"

To get pandas to import at run-time correctly I had to modify the Hello.spec to the following:

# -*- mode: python -*-

block_cipher = None

def get_pandas_path():
    import pandas
    pandas_path = pandas.__path__[0]
    return pandas_path

a = Analysis(['Hello.py'],
         pathex=['C:\\ScriptsThatRequirePandas'],
         binaries=None,
         datas=None,
         hiddenimports=[],
         hookspath=None,
         runtime_hooks=None,
         excludes=None,
         win_no_prefer_redirects=None,
         win_private_assemblies=None,
         cipher=block_cipher)

dict_tree = Tree(get_pandas_path(), prefix='pandas', excludes=["*.pyc"])
a.datas += dict_tree
a.binaries = filter(lambda x: 'pandas' not in x[0], a.binaries)

pyz = PYZ(a.pure, a.zipped_data,
         cipher=block_cipher)
exe = EXE(pyz,
      a.scripts,
      exclude_binaries=True,
      name='Hello',
      debug=False,
      strip=None,
      upx=True,
      console=True )
scoll = COLLECT(exe,
           a.binaries,
           a.zipfiles,
           a.datas,
           strip=None,
           upx=True,
           name='Hello')

I then ran:

$pyinstaller Hello.spec --onefile

from the command prompt and got the 'hello world' message I expected. I still don't completely understand why this is necessary. I have a custom build of pandas - which is hooked into the MKL libraries - but it isn't clear to me that this is causing the run failure.

This is similar to the answer here: Pyinstaller not correclty importing pycripto... sometimes

Answer 2

I had a similar issue with pyinstaller version 3.3. The solution was that there was a missing hiddenimport hook as described here

I created a new file under Pyinstaller/hooks/ called hook-pandas.py and put the content as described in this commit here and reinstalled pyinstaller manually via python setup.py install in the Pyinstaller directory.

The issue did not recur when I created exe from my pandas script with pyinstaller using the --onefile option.

Answer 3

Just as another solution, adding --hidden-import=pandas._libs.tslibs.timedelta or whatever the module is missing to the pyinstaller command also works.

This may be helpful if you don't want to touch the source of pyinstaller.

Answer 4

I had exactly the same problem, and I found another solution (the only one that worked for me actually) :

I followed pretty much this : https://medium.com/@liron92/pyinstaller-with-pandas-problems-solutions-and-workflow-with-code-examples-c72973e1e23f

Except for the fact that I run my virtual environnement on Anaconda.

Quick step-by-step sollution

Before I start : these are the steps that I followed for my particuliar case, you might want to adapt a little bit depending on your situation.

1) Create and configure your virtual environnement

I used Anaconda to create my env :

conda create --name myenv

Then I installed all the module I needed :

conda install -n myenv pandas
conda install -n myenv -c conda-forge python-docx
etc.

2) Activate your environnement and switch to project path

On Anaconda Prompt :

conda activate myenv
cd path/to/your/project/folder

3) Create and modify your *.spec file

Still on the same Anaconda Prompt window :

pyi-makespec project.py

Then open your project.spec file, it will looks like that:

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['project.py'],
             pathex=['path/to/your/project/folder'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='main',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='main')

You just modify hiddenimports = [] and add all the implicit import (which include pandas). In my case, I was also using Tkinter, so I specified :

hiddenimports=['pandas', 'tkinter']

4) Finally run pyinstaller

On the same Anaconda Prompt window (environnement activated, on your project directory) :

pyinstaller main.spec

And then you're done !

Disclaimer

I see everywhere people telling you that you should use --onefile when you compile a Python project with Pyinstaller, I honestly think you shouldn't : it makes the *.exe way slower. Maybe I'm missing something with this so please if you use it explain to me.

Specs

Windows 10
Anaconda 4.8.2
Python 3.7.6
Pandas 1.0.5