Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Cython attemps to compile twice, and fails

I have a setup.py file that is very similar to the one shown here: https://stackoverflow.com/a/49866324/4080129. It looks like this:

from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy

sources = ["hs/subfolder/detect.pyx",
           "hs/subfolder/Donline.cpp",
           "hs/subfolder/Handler.cpp",
           "hs/subfolder/Process.cpp",
           "hs/subfolder/Filter.cpp",
           "hs/subfolder/Localize.cpp"]

exts = [Extension(name='hs.detect',
                  sources=sources,
                  extra_compile_args=['-std=c++11', '-O3'],
                  include_dirs=[numpy.get_include()])]

setup(
    ext_modules=cythonize(exts),
    include_dirs=[numpy.get_include()]
)

There's a package with some pure-Python, and a submodule that contains Cython files. The setup.py is in the parent folder, not in the Cython one:

setup.py
hs/
    some_python.py
    subfolder/
        detect.pyx
        Donline.cpp
        ...etc

Now, setup.py correctly compiles all the files module/submodule/file1.cpp etc. and saves the build to build/temp.linux-x86_64-3.6/module/submodule/file1.o . However, just after that, it tries to compile a file called file1.cpp, which doesn't exist (the correct one is module/submodule/file1.cpp, and has already been compiled).

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ihs/subfolder -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/include -I/disk/scratch/miniconda/envs/my_default/include/python3.6m -c Donline.cpp -o build/temp.linux-x86_64-3.6/Donline.o -std=c++11 -O3
gcc: error: Donline.cpp: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4

I'm very confused, this completely prevents my code from compiling...

like image 611
Martino Avatar asked May 21 '18 11:05

Martino


1 Answers

It turns out the .pyx file contains a line

# distutils: sources = Donline.cpp Handler.cpp Process.cpp Filter.cpp Localize.cpp

which tells distutils what to compile. I wasn't aware of it, and since it looks an awful lot like a commented-out line, I didn't realise it was there.

Cython tries to compile also these, other than the ones contained in the setup.pyfile, i.e. neither of the two sources list overrides the other. Apparently, these sources, despite being listed in the pyx file, which is in a subfolder, are expected to be in paths relative to the file where the setup.py file is, or perhaps relative to the folder I'm calling python from.

Anyway, removing the line solved the issue.

like image 119
Martino Avatar answered Oct 23 '22 07:10

Martino