Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Importing any module in cython file gives undefined symbol error

When I use any import statement within modules I compile with cython, I receive the following error on importing the modules (see full code below):

ImportError: /.../hw.cpython-35m-x86_64-linux-gnu.so: undefined symbol: __intel_sse2_strchr

Everything works fine on my own machine, but I get this error when trying to recompile and run my scripts on an external high-performance computer.

Googling I saw similar errors pop up in several places, but in these discussions the problem is either related to an error in the own code, rather than an imported module (e.g. Undefined symbol error importing Cython module) or something goes wrong at building cython or compile time (e.g. Can cython be compiled with icc?). I don't see how to apply these to my case.

From reading other discussions, I suspected the problem was in the formulation of my setup.py, but I really don't know much about that - on my own system the standard examples simply worked. I tried adding numpy to the setup (see setup2.py below), but that didn't solve the problem - I get the same error on import hw.

Full example

hw.pyx:

import numpy  # without this line, the example works fine 

cpdef testfun():
    print("Hello world!")

setup.py:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize('hw.pyx'),
)

setup2.py:

from distutils.core import setup
from Cython.Build import cythonize
import numpy as np

setup(
    ext_modules = cythonize('hw.pyx'),
    include_dirs = [np.get_include()]  # -> including this doesn't make a difference
)

System info

The external system runs python 3.5.2. I run my code within a virtualenv, where I have Cython 0.28.1. I tried both numpy 1.13.0 and 1.14.2. For the virtuel environment in this example, I did not install any other packages (probably not related to my problem, but pip freeze also lists UNKNOWN==0.0.0)

[IMPORTANT EDIT]

I received the error for numpy (and since I had struggled before with cdef np.ndarray I figured that was the problem) - but actually it is any import statement that causes the error - numpy just happened to be the first module that was imported. Still no clue how to resolve this though.

like image 638
Scipio Avatar asked Nov 08 '22 08:11

Scipio


1 Answers

The problem was that I used the Intel compiler without running the Intel Python distribution, or otherwise providing the Intel runtime libraries.

Thanks to @ead, I resolved the problem by switching to GCC using:

CC=gcc python setup.py build_ext --inplace
like image 168
Scipio Avatar answered Nov 14 '22 23:11

Scipio