Is there *any* solution to packaging a python app that uses cppyy?

Question

I'm no novice when creating cross-platform runtimes of my python desktop apps. I create various tools for my undergraduates using mostly pyinstaller, cxfreeze, sometimes fbs, and sometimes briefcase. Anyone who does this one a regular basis knows that there are lots of quirks and adjustments needed to target Linux, windows, and macos when using arbitrary collections of python modules, but I've managed to figure everything out until now.

I have a python GUI app that uses a c++ library that is huge and ever-changing, so I can't just re-write it in python. I've successfully written python code that uses the c++ library using the amazing (and possibly magical) library called cppyy that allows you to run c++ code from python without hardly any effort. Everything runs great on Linux, mac, and windows, but I cannot get it packaged into runtimes and I've tried all the systems above. All of them have no problem producing the runtimes (i.e., no errors), but they fail when you run them. Essentially they all give some sort of error about not being able to find cppyy-backend (e.g., pyinstaller and fbs which uses pyinstaller gives this message when you run the binary):

/home/nogard/Desktop/cppyytest/target/MyApp/cppyy_backend/loader.py:113: UserWarning: No precompiled header available ([Errno 2] No such file or directory: '/home/nogard/Desktop/cppyytest/target/MyApp/cppyy_backend'); this may impact performance.
Traceback (most recent call last):
  File "main.py", line 5, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "/home/nogard/Desktop/cppyytest/venv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 628, in exec_module
    exec(bytecode, module.__dict__)
  File "cppyy/__init__.py", line 74, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "/home/nogard/Desktop/cppyytest/venv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 628, in exec_module
    exec(bytecode, module.__dict__)
  File "cppyy/_cpython_cppyy.py", line 20, in <module>
  File "cppyy_backend/loader.py", line 74, in load_cpp_backend
RuntimeError: could not load cppyy_backend library
[11195] Failed to execute script main

I'm really stumped. Usually, you install cppyy with pip, which installs cppyy-backend and other packages. I've even used the cppyy docs methods to compile each dependency as well as cppyy, but the result is the same.

I'll use any build system that works...has anyone had success? I know I could use docker, but I tried this before and many of my students freaked out at docker asking them to change their bios settings to support virtualization So I'd like to use a normal packaging system that produces some sort of runnable binary.

If you know how to get pyinstaller, cxfreeze, fbs, or briefcase to work with cppyy (e.g, if you know how to deal with the error above), please let me know. However, if you've gotten a cppyy app packaged with some other system, let me know and I'll use that one.

If you're looking for some code to run, I've been testing out packaging methods using this minimal code:

import cppyy

print('hello world from python\n')

cppyy.cppexec('''
#include <string>
using namespace std;
string mystring("hello world from c++");
std::cout << mystring << std::endl;
''')

Answer 1

EDIT: figured out the pyinstaller hooks; this should all be fully automatic once released

With the caveat that I have no experience whatsoever with packaging run-times, so I may be missing something obvious, but I've just tried pyinstaller, and the following appears to work.

First, saving your script above as example.py, then create a spec file:

$ pyi-makespec example.py

Then, add the headers and libraries from cppyy_backend as datas (skipping the python files, which are added by default). The simplest seems to be to pick up all directories from the backend, so change the generated example.spec by adding at the top:

def backend_files():
    import cppyy_backend, glob, os

    all_files = glob.glob(os.path.join(
        os.path.dirname(cppyy_backend.__file__), '*'))

    def datafile(path):
        return path, os.path.join('cppyy_backend', os.path.basename(path))

    return [datafile(filename) for filename in all_files if os.path.isdir(filename)]

and replace the empty datas in the Analysis object with:

             datas=backend_files(),

If you also need the API headers from CPyCppyy, then these can be found e.g. like so:

def api_files():
    import cppyy, os

    paths = str(cppyy.gbl.gInterpreter.GetIncludePath()).split('-I')
    for p in paths:
        if not p: continue
       
        apipath = os.path.join(p.strip()[1:-1], 'CPyCppyy')
        if os.path.exists(apipath):
            return [(apipath, os.path.join('include', 'CPyCppyy'))]

    return []

and added to the Analysis object:

             datas=backend_files()+api_files(),

Note however, that Python.h then also needs to exist on the system where the package will be deployed. If need be, Python.h can be found through module sysconfig and its path provided through cppyy.add_include_path in the bootstrap.py file discussed below.

Next, consider the precompiled header (file cppyy_backend/etc/allDict.cxx.pch): this contains the C++ standard headers in LLVM intermediate representation. If addded, it pre-empts the need for a system compiler where the package is deployed. However, if there is a system compiler, then ideally, the PCH should be recreated on first use after deployment.

As is, however, the loader.py script in cppyy_backend uses sys.executable which is broken by the freezing (meaning, it's the top-level script, not python, leading to an infinite recursion). And even when the PCH is available, its timestamp is compared to the timestamp of the include directory, and rebuild if older. Since both the PCH and the include directory get new timestamps based on copy order, not build order, this is unreliable and may lead to spurious rebuilding. Therefore, either disable the PCH, or disable the time stamp checking.

To do so, choose one of these two options and write it in a file called bootstrap.py, by uncommenting the desired behavior:

### option 1: disable the PCH altogether

# import os
# os.environ['CLING_STANDARD_PCH'] = 'none'

### option 2: force the loader to declare the PCH up-to-date

# import cppyy_backend.loader
#
# def _is_uptodate(*args):
#    return True
#
# cppyy_backend.loader._is_uptodate = _is_uptodate

then add the bootstrap as a hook to the spec file in the Analysis object:

             runtime_hooks=['bootstrap.py'],

As discussed above, the bootstrap.py is also a good place to add more include paths as necessary, e.g. for Python.h.

Finally, run as usual:

$ pyinstaller example.spec