Python - C embedded Segmentation fault

Question

I am facing a problem similar to the Py_initialize / Py_Finalize not working twice with numpy .. The basic coding in C:

Py_Initialize();
import_array();
//Call a python function which imports numpy as a module
//Py_Finalize()

The program is in a loop and it gives a seg fault if the python code has numpy as one of the imported module. If I remove numpy, it works fine.

As a temporary work around I tried not to use Py_Finalize(), but that is causing huge memory leaks [ observed as the memory usage from TOP keeps on increasing ]. And I tried but did not understand the suggestion in that link I posted. Can someone please suggest the best way to finalize the call while having imports such as numpy.

Thanks santhosh.

Answer 1

I recently faced a very similar issue and developed a workaround that works for my purposes, so I thought I would write it here in the hope it might help others.

The problem

I work with some postprocessing pipeline for which I can write a own functor to work on some data passing through the pipeline and I wanted to be able to use Python scripts for some of the operations.

The problem is that the only thing I can control is the functor itself, which gets instantiated and destroyed at times beyond my control. I furthermore have the problem that even if I do not call Py_Finalize the pipeline sometimes crashes once I pass another dataset through the pipeline.

The solution in a Nutshell

For those who don't want to read the whole story and get straight to the point, here's the gist of my solution:

The main idea behind my workaround is not to link against the Python library, but instead load it dynamically using dlopen and then get all the addresses of the required Python functions using dlsym. Once that's done, one can call Py_Initialize() followed by whatever you want to do with Python functions followed by a call to Py_Finalize() once you're done. Then, one can simply unload the Python library. The next time you need to use Python functions, simply repeat the steps above and Bob's your uncle.

However, if you are importing NumPy at any point between Py_Initialize and Py_Finalize, you will also need to look for all the currently loaded libraries in your program and manually unload those using dlclose.

Detailed workaround

Loading instead of linking Python

The main idea as I mentioned above is not to link against the Python library. Instead, what we will do is load the Python library dynamically using dlopen():

#include ... void* pHandle = dlopen("/path/to/library/libpython2.7.so", RTLD_NOW | RTLD_GLOBAL);

The code above loads the Python shared library and returns a handle to it (the return type is an obscure pointer type, thus the void*). The second argument (RTLD_NOW | RTLD_GLOBAL) is there to make sure that the symbols are properly imported into the current application's scope.

Once we have a pointer to the handle of the loaded library, we can search that library for the functions it exports using the dlsym function:

#include <dlfcn.h>
...
// Typedef named 'void_func_t' which holds a pointer to a function with
// no arguments with no return type
typedef void (*void_func_t)(void);
void_func_t MyPy_Initialize = dlsym(pHandle, "Py_Initialize");

The dlsym function takes two parameters: a pointer to the handle of the library that we obtained previously and the name of the function we are looking for (in this case, Py_Initialize). Once we have the address of the function we want, we can create a function pointer and initialize it to that address. To actually call the Py_Initialize function, one would then simply write:

MyPy_Initialize();

For all the other functions provided by the Python C-API, one can just add calls to dlsym and initialize function pointers to its return value and then use those function pointers instead of the Python functions. One simply has to know the parameter and return value of the Python function in order to create the correct type of function pointer.

Once we are finished with the Python functions and call Py_Finalize using a procedure similar to the one for Py_Initialize one can unload the Python dynamic library in the following way:

dlclose(pHandle);
pHandle = NULL;

Manually unloading NumPy libraries

Unfortunately, this does not solve the segmentation fault problems that occur when importing NumPy. The problems comes from the fact that NumPy also loads some libraries using dlopen (or something equivalent) and those do not get unloaded them when you call Py_Finalize. Indeed, if you list all the loaded libraries within your program, you will notice that after closing the Python environment with Py_Finalize, followed by a call to dlclose, some NumPy libraries will remain loaded in memory.

The second part of the solution requires to list all the Python libraries that remain in memory after the call dlclose(pHandle);. Then, for each of those libraries, grab a handle to them and then call dlcloseon them. After that, they should get unloaded automatically by the operating system.

Fortunately, there are functions under both Windows and Linux (sorry MacOS, couldn't find anything that would work in your case...): - Linux: dl_iterate_phdr - Windows: EnumProcessModules in conjunction with OpenProcess and GetModuleFileNameEx

Linux

This is rather straight forward once you read the documentation about dl_iterate_phdr:

#include <link.h>
#include <string>
#include <vector>

// global variables are evil!!! but this is just for demonstration purposes...
std::vector<std::string> loaded_libraries;

// callback function that gets called for every loaded libraries that
// dl_iterate_phdr finds
int dl_list_callback(struct dl_phdr_info *info, size_t, void *)
{
    loaded_libraries.push_back(info->dlpi_name);
    return 0;
}

int main()
{
    ...
    loaded_libraries.clear();
    dl_iterate_phdr(dl_list_callback, NULL);
    // loaded_libraries now contains a list of all dynamic libraries loaded
    // in your program
    ....
}

Basically, the function dl_iterate_phdr cycles through all the loaded libraries (in the reverse order they were loaded) until either the callback returns something other than 0 or it reaches the end of the list. To save the list, the callback simply adds each element to a global std::vector (one should obviously avoid global variables and use a class for example).

Windows

Under Windows, things get a little more complicated, but still manageable:

#include <windows.h>
#include <psapi.h>

std::vector<std::string> list_loaded_libraries()
{
     std::vector<std::string> m_asDllList;
     HANDLE hProcess(OpenProcess(PROCESS_QUERY_INFORMATION 
                                 | PROCESS_VM_READ,
                                 FALSE, GetCurrentProcessId()));
     if (hProcess) {
         HMODULE hMods[1024];
         DWORD cbNeeded;

         if (EnumProcessModules(hProcess, hMods, sizeof(hMods), &cbNeeded)) {
             const DWORD SIZE(cbNeeded / sizeof(HMODULE));
             for (DWORD i(0); i < SIZE; ++i) {
                TCHAR szModName[MAX_PATH];

                // Get the full path to the module file.
                if (GetModuleFileNameEx(hProcess,
                                        hMods[i],
                                        szModName,
                                        sizeof(szModName) / sizeof(TCHAR))) {
#ifdef UNICODE
                    std::wstring wStr(szModName);
                    std::string tModuleName(wStr.begin(), wStr.end());
#else
                    std::string tModuleName(szModName);
#endif /* UNICODE */
                    if (tModuleName.substr(tModuleName.size()-3) == "dll") {
                        m_asDllList.push_back(tModuleName);
                    }
                 }
             }
         }
         CloseHandle(hProcess);
     }
     return m_asDllList;
}

The code in this case is slightly longer than for the Linux case, but the main idea is the same: list all the loaded libraries and save them into a std::vector. Don't forget to also link your program to the Psapi.lib!

Manual unloading

Now that we can list all the loaded libraries, all you need to do is find among those the ones that come from loading NumPy, grab a handle to them and then call dlclose on that handle. The code below will work on both Windows and Linux, provided that you use the dlfcn-win32 library.

#ifdef WIN32
#  include <windows.h>
#  include <psapi.h>
#  include "dlfcn_win32.h"
#else
#  include <dlfcn.h>
#  include <link.h> // for dl_iterate_phdr
#endif /* WIN32 */

#include <string>
#include <vector>

// Function that list all loaded libraries (not implemented here)
std::vector<std::string> list_loaded_libraries();


int main()
{
    // do some preprocessing stuff...

    // store the list of loaded libraries now
    // any libraries that get added to the list from now on must be Python
    // libraries
    std::vector<std::string> loaded_libraries(list_loaded_libraries());
    std::size_t start_idx(loaded_libraries.size());

    void* pHandle = dlopen("/path/to/library/libpython2.7.so", RTLD_NOW | RTLD_GLOBAL);

    // Not implemented here: get the addresses of the Python function you need

    MyPy_Initialize(); // Needs to be defined somewhere above!

    MyPyRun_SimpleString("import numpy"); // Needs to be defined somewhere above!

    // ...

    MyPyFinalize(); // Needs to be defined somewhere above!

    // Now list the loaded libraries again and start manually unloading them
    // starting from the end
    loaded_libraries = list_loaded_libraries();

    // NB: this below assumes that start_idx != 0, which should always hold true
    for(std::size_t i(loaded_libraries.size()-1) ; i >= start_idx ; --i) {
        void* pHandle = dlopen(loaded_libraries[i].c_str(),
#ifdef WIN32
                               RTLD_NOW // no support for RTLD_NOLOAD
#else
                               RTLD_NOW|RTLD_NOLOAD                   
#endif /* WIN32 */
                    );
        if (pHandle) {
            const unsigned int Nmax(50); // Avoid getting stuck in an infinite loop
            for (unsigned int j(0) ; j < Nmax && !dlclose(pHandle) ; ++j);
        }
    }
}

Final words

The examples shown here capture the basic ideas behind my solution, but can certainly be improved to avoid global variables and facilitate ease of use (for example, I wrote a singleton class that handles the automatic initialization of all the function pointers after loading the Python library).

I hope this can be useful to someone in the future.

References

• dl_iterate_phdr: https://linux.die.net/man/3/dl_iterate_phdr
• PsAPI library: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684894(v=vs.85).aspx
• OpenProcess: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684320(v=vs.85).aspx
• EnumProcess: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682629(v=vs.85).aspx
• GetModuleFileNameEx: https://msdn.microsoft.com/en-us/library/windows/desktop/ms683198(v=vs.85).aspx
• dlfcn-win32 library: library: https://github.com/dlfcn-win32/dlfcn-win32

Answer 2

I'm not quite sure how you don't seem to understand the solution posted in Py_initialize / Py_Finalize not working twice with numpy. The solution posted is quite simple: call Py_Initialize and Py_Finalize only once for each time your program executes. Do not call them every time you run the loop.

I assume that your program, when it starts, runs some initialization commands (which are only run once). Call Py_Initialize there. Never call it again. Also, I assume that when your program terminates, it has some code to tear down things, dump log files, etc. Call Py_Finalize there. Py_Initialize and Py_Finalize are not intended to help you manage memory in the Python interpreter. Do not use them for that, as they cause your program to crash. Instead, use Python's own functions to get rid of objects you don't want to keep.

If you really MUST create a new environment every time you run your code, you can use Py_NewInterpreter and to create a sub-interpreter and Py_EndInterpreter to destroy that sub-interpreter later. They're documented near the bottom of the Python C API page. This works similarly to having a new interpreter, except that modules are not re-initialized each time a sub-interpreter starts.