How to compile my python code in cython with external python libs like pybrain

Question

I need more perfomance running my neural network, so I thinked that building it with cython will be good idea. I am building my code like this:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("my_code.pyx")
)

But will it build external python files that I use? Like pybrain, skimage and PIL in my case. If not, how to force cython to build them.

Answer 1

No, external python files will not be cythonized and compiled unless you specifically add them to your setup.py as an extension. As far as I know there is no trivial way to do this.

This means that all calls to the external files will be handled in 'Python-space' and hence can not use the full potential of Cython. For example all calls to an external file will be type checked, which wastes a lot of time. You can see this if you cythonize a file using cython -a yourfile.pyx and take a look at the created C code. The more yellow there is the more pythony your code is.

You have the following options:

1. Find libraries / packages that offer Cython or C-level access. Unfortunately chances are low that you will find good ones (or any at all) using Cython and building a wrapper for a C library is a lot of work. Note that packages that themselve are implemented in C (like numpy for example) already are reasonably fast. I do not know how this behaves with your packages in question. pybrains seems to be pure python from what I saw at first glance.
2. Get the source code of the packages you want to use and compile them yourself with Cython. This might be an awful lot of work and not worth the time.
3. Find the bottlenecks using a profiler like lineprofiler / kernprof (this should always be the first step when optimizing) and try to cythonize only the runtime bottlenecks.

I personally would go with option three, as options one and two both might require a lot of work on your side with questionable outcome.