Runtime generation and compilation of Cython functions

Question

A brief version

Is there an easy way to compile a Cython function in runtime given the code of the function as a string?

In details

I have a parametrized subroutine e.g.

cdef algo(x, params)

• the algorithm performs a relatively small number of operations, but is called very frequently;
• params are not known in compile time, but are known at the beginning (e.g. set via config) and are fixed for the whole life time of the program;
• algo can be algorithmically optimized for a specific params (optimizations go beyond what a compiler can do), however, the number of possible optimized algo is extremely large.

In other words, there is a function that receives params and yields a code of a fast implementation of algo for these params:

def meta_algo(params):
  <meta magic>
  return code_of_super_fast_algo

The question is how one can compile and import the function defined by the output of meta_algo?

An example

Suppose that you have a fixed small set of strings ys. For a given another string x you want to compute length of the maximal common prefix of x with each string from ys and return it as an array of integers. A reference naive implementation:

def max_prefix(x, ys):
  result = []
  for i, y in enumerate(ys):
    j = 0
    while x[j] == y[j]:
      j++

    result[i] = j

  return result

For example, if known that the strings of ys heavily intersect with each other, one can easily compute a comparison tree, but using any tree data-structure will introduce an undesirable overhead. Instead, one can 'inline' this tree structure in a series of ifs and generate an efficient code.

For ys = ['aaa', 'aab', 'baa'] one might get:

cdef max_prefix(str x):
  if x[0] == 'a':
    if x[1] != 'a':
      return [1, 1, 0]
    if x[2] == 'a':
      return [3, 2, 0]
    elif x[2] == 'b':
      return [2, 3, 0]
    else:
      return [2, 2, 0]
  elif ...:
    ...

Answer 1

After digging a little bit deeper into Cython guts, I have found the following pool request:

https://github.com/cython/cython/pull/555

which provide the exact the functionality I wanted:

code = """
cpdef int plus(int a, int b):
  return a + b
"""

module = cython_inline_module(code)
plus = getattr(module, 'plus')

assert plus(1, 2) == 3