Link f2py generated *.so file in a python package using setuptools

Question

I wish to deploy a package to PyPi using setuptools. However, the core part of the package is actually written in Fortran, and I am using f2py to wrap it in python. Basically the project's structure looks like this:

my_project

• license.txt
• README.md
• setup.py
• my_project
  • init.py
  • myfunc.py
  • hello.so

The module myfunc.py imports hello.so (import my_project.hello) which can then be used by functions inside myfunc.py. This works perfectly on my machine.

Then I tried standard setuptools installation: sudo python3 setup.py install on my Ubuntu, and it gets installed perfectly. But unfortunately, while importing, it throws ModuleNotFoundError: No module named 'hello'.

Now, from what I understand, on Linux based systems, for python, the shared libraries *.so are stored in /usr/lib/python3/dist-packages/. So I manually copied this hello.so there, and I got a working package! But of course this works only locally. What I would like to do is to tell setuptools to include hello.so inside the python-egg and automatically do the copying etc so that when a user uses pip3 install my_package, they will have access to this shared library automatically. I can see that numpy has somehow achieved that but even after looking at their code, I haven't been able to decode how they did it. Can someone help me with this? Thanks in advance.

Answer 1

You can achieve this with a setup.py file like this (simplified version, keep only the relevant parts for building external modules)

import os
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext


class f2py_Extension(Extension):

    def __init__(self, name, sourcedirs):
        Extension.__init__(self, name, sources=[])
        self.sourcedirs = [os.path.abspath(sourcedir) for sourcedir in sourcedirs]
        self.dirs = sourcedirs

class f2py_Build(build_ext):

    def run(self):
        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        # compile
        for ind,to_compile in enumerate(ext.sourcedirs):
            module_loc = os.path.split(ext.dirs[ind])[0]
            module_name = os.path.split(to_compile)[1].split('.')[0]
            os.system('cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name))

setup(
    name="foo",
    ext_modules=[f2py_Extension('fortran_external',['foo/one.F90','foo/bar/two.F90'])],
    cmdclass=dict(build_ext=f2py_Build),
)

The essential parts for building an external module are ext_modules and cmdclass in setup(...). ext_modules is just a list of Extension instances, each of which describes a set of extension modules. In the setup.py above, I tell ext_modules I want to create two external modules with two source files foo/test.F90 and foo/bar/two.F90. Based on ext_modules, cmdclass is responsible for compiling the two modules, in our case, the command for compiling the module is

'cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name)

---

Project structure before installation

├── foo
│   ├── __init__.py
│   ├── bar
│   │   └── two.F90
│   └── one.F90
└── setup.py

Project structure after python setup.py install

├── build
│   └── bdist.linux-x86_64
├── dist
│   └── foo-0.0.0-py3.7-linux-x86_64.egg
├── foo
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-37.pyc
│   ├── bar
│   │   ├── two.F90
│   │   └── two.cpython-37m-x86_64-linux-gnu.so
│   ├── one.F90
│   └── one.cpython-37m-x86_64-linux-gnu.so
├── foo.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
└── setup.py

The two source files one.F90 and two.F90 are very simple

one.F90

module test

  implicit none

  contains

  subroutine add(a)

    implicit none
    integer :: a
    integer :: b
    b = a + 1
    print *, 'one',b

  end subroutine add


end module test

two.F90

module test

  implicit none

  contains

  subroutine add(a)

    implicit none
    integer :: a
    integer :: b
    b = a + 2
    print *, 'two',b

  end subroutine add


end module test

After I installed the package, I can successfully run

>>> from foo.bar.two import test
>>> test.add(5)
 two           7

and

>>> from foo.one import test
>>> test.add(5)
 one           6