Executable made with pyInstaller/UPX experiences QtCore4.dll error

Question

A python program, which I compiled with pyInstaller, turned out to be over 400 MB. The program's GUI is based on htmlPY, which is "a wrapper around PySide's QtWebKit library." The large size of the program partly owes to the fact that it utilizes numpy, scipy, and nltk, and in part due to the graphics libraries.

To minimize the size of the program, I installed UPX. This decreased the size of the program to slightly over 100MB, which is large, but acceptable.

The first problem is that pyInstaller didn't detect htmlPy, and didn't include it in the compiled program. This can be fixed by copying the htmlPy module from my Python installation, into the 'dist' directory created by pyInstaller. After doing this, the version of the program compiled without UPX, was working fine.

After adding htmlPy to the 'dist' directory, running the executable crashes the program at the point when the GUI is created. I'm not sure if this is due to a problematic interaction between UPX and QT, or between UPX, QT, and htmlPy. The Windows "Problem Signature" is the following:

Problem signature:
  Problem Event Name:   APPCRASH
  Application Name: main.exe
  Application Version:  0.0.0.0
  Application Timestamp:    00000000
  Fault Module Name:    QtCore4.dll
  Fault Module Version: 4.8.7.0
  Fault Module Timestamp:   561e435a
  Exception Code:   c0000005
  Exception Offset: 000000000010883a

Any ideas as to what's going on here, and how to fix it?

EDIT:

These are the contents of my .spec file:

# -*- mode: python -*-

block_cipher = None

added_files = [
     ( 'htmlPy/binder.js', 'htmlPy' ),
     ( 'templates/*', 'templates' ),
   ]
a = Analysis(['main.py'],
             pathex=['C:\\..\\My_App'],
             binaries=None,
             datas=added_files,
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=['rthook_pyqt4.py'],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='My_App',
          debug=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='My_App')

These are the contents of rthook_pyqt4.py:

import sip

sip.setapi(u'QDate', 2)
sip.setapi(u'QDateTime', 2)
sip.setapi(u'QString', 2)
sip.setapi(u'QTextStream', 2)
sip.setapi(u'QTime', 2)
sip.setapi(u'QUrl', 2)
sip.setapi(u'QVariant', 2)

Edit 2:

Here's some of the initialization code (standard htmlPy fare):

app.static_path = path.join(BASE_DIR, "static/")
print "Step 1"
app.template_path = path.join(BASE_DIR, "templates/")
print "Step 2"
app.template = ("index.html", {"username": "htmlPy_user"})
print "Step 3"
...

The program crashes before it gets to Step 3.

Answer 1

Your two big concerns relate to:

1. correctness - app with UPX won't run
2. performance - 400 MiB is "too big" and 100 MiB lets you address a bigger set of users

The app might be more useful to more people if it's smaller, but it's useful to no one if it won't run. You suspect that UPX improves concern 2 but its interactions impact concern 1.

It would be interesting to build a simple HelloWorld app, package it with pyInstaller + UPX, and keep embellishing it with additional dependencies (like Qt) until you see it break in a way like the current breakage.

It might be more productive to abandon UPX in favor of other approaches, including NSIS. You might use a tool like strace() to monitor which of your distributed files are actually used during system testing runs, and prune unused files during packaging. Proxying requests through FUSE would yield similar information. You might list dependencies for your published app, and rely on pip or conda to download dependencies in parallel, if "elapsed install time" is really what drives your desire to shrink 400 down to 100 MiB.