Cython: pyximport: enabling profiling in pyximport.install

Question

My Python 3 projects are using cython a lot.

In production deployment, I'm using a build script which, amongst other things, disables the profiling:

from distutils.core import setup
from Cython.Build import cythonize
import os

compiler_directives = {
    'language_level': 3,
    'optimize.use_switch': True,
    'profile': True,
}

setup(
    packages=["XXXXXX"],
    ext_modules=cythonize(
        module_list="**/*.pyx",
        compiler_directives=compiler_directives,
    )
)

In development, I'm using pyximport. To make the difference between the two contexts, I'm testing if the "production" user is in use in the project's top level __init__.py file. If this is not production, I'm using pyximport; pyximport.install, so that it becomes totally transparent:

if getpass.getuser != PRODUCTION_USER_NAME:
    import pyximport
    pyximport.install(
        pyximport=True,
        pyimport=False,
        build_dir=None,
        build_in_temp=True,
        setup_args={},
        reload_support=False,
        load_py_module_on_import_failure=False,
        inplace=False,
        language_level=3,
    )

I'm would like to enable the profiling for all cython files while in development environment. I've tried to add the profile=True argument to the piximport.install statement, but it doesn't works.

How can I proceed?

Some additional comments:

• I'd like to avoid pushing Profile=True in all the source codes while in development and removing them before commiting...

• Using .pyxbld files is not an option for me because I have 46 pyx files and plan to have many more... Unless there's a way to setup only one file to support all the pyx like I did for the build script, but I didn't found how.

Thanks for your help.

Answer 1

It requires wrapping one of pyximport's internal functions, but this can be done:

# Allow .pyx files to be seamlessly integrated via cython/pyximport with
# default compiler directives.
import functools
import pyximport.pyximport

# Hack pyximport to have default options for profiling and embedding signatures
# in docstrings.
# Anytime pyximport needs to build a file, it ends up calling
# pyximport.pyximport.get_distutils_extension.  This function returns an object
# which has a cython_directives attribute that may be set to a dictionary of
# compiler directives for cython.
_old_get_distutils_extension = pyximport.pyximport.get_distutils_extension
@functools.wraps(_old_get_distutils_extension)
def _get_distutils_extension_new(*args, **kwargs):
    extension_mod, setup_args = _old_get_distutils_extension(*args, **kwargs)

    if not hasattr(extension_mod, 'cython_directives'):
        extension_mod.cython_directives = {}
    extension_mod.cython_directives.setdefault('embedsignature', True)
    extension_mod.cython_directives.setdefault('profile', True)
    return extension_mod, setup_args
pyximport.pyximport.get_distutils_extension = _get_distutils_extension_new
pyximport.install()

Note that this will not forcibly recompile unchanged modules with the new options; you will have to touch those files to trigger a compilation with the new configuration.