How can I unload a DLL using ctypes in Python?

Question

I'm using ctypes to load a DLL in Python. This works great.

Now we'd like to be able to reload that DLL at runtime.

The straightforward approach would seem to be: 1. Unload DLL 2. Load DLL

Unfortunately I'm not sure what the correct way to unload the DLL is.

_ctypes.FreeLibrary is available, but private.

Is there some other way to unload the DLL?

Answer 1

you should be able to do it by disposing the object

mydll = ctypes.CDLL('...')
del mydll
mydll = ctypes.CDLL('...')

EDIT: Hop's comment is right, this unbinds the name, but garbage collection doesn't happen that quickly, in fact I even doubt it even releases the loaded library.

Ctypes doesn't seem to provide a clean way to release resources, it does only provide a _handle field to the dlopen handle...

So the only way I see, a really, really non-clean way, is to system dependently dlclose the handle, but it is very very unclean, as moreover ctypes keeps internally references to this handle. So unloading takes something of the form:

mydll = ctypes.CDLL('./mylib.so')
handle = mydll._handle
del mydll
while isLoaded('./mylib.so'):
    dlclose(handle)

It's so unclean that I only checked it works using:

def isLoaded(lib):
   libp = os.path.abspath(lib)
   ret = os.system("lsof -p %d | grep %s > /dev/null" % (os.getpid(), libp))
   return (ret == 0)

def dlclose(handle)
   libdl = ctypes.CDLL("libdl.so")
   libdl.dlclose(handle)

Answer 2

It is helpful to be able to unload the DLL so that you can rebuild the DLL without having to restart the session if you are using iPython or similar work flow. Working in windows I have only attempted to work with the windows DLL related methods.

REBUILD = True
if REBUILD:
  from subprocess import call
  call('g++ -c -DBUILDING_EXAMPLE_DLL test.cpp')
  call('g++ -shared -o test.dll test.o -Wl,--out-implib,test.a')

import ctypes
import numpy

# Simplest way to load the DLL
mydll = ctypes.cdll.LoadLibrary('test.dll')

# Call a function in the DLL
print mydll.test(10)

# Unload the DLL so that it can be rebuilt
libHandle = mydll._handle
del mydll
ctypes.windll.kernel32.FreeLibrary(libHandle)

I don't know much of the internals so I'm not really sure how clean this is. I think that deleting mydll releases the Python resources and the FreeLibrary call tells windows to free it. I had assumed that freeing with FreeLibary first would have produced problems so I saved a copy of the library handle and freed it in the order shown in the example.

I based this method on ctypes unload dll which loaded the handle explicitly up front. The loading convention however does not work as cleanly as the simple "ctypes.cdll.LoadLibrary('test.dll')" so I opted for the method shown.

Answer 3

windows and linux compatible minimal reproducible example from 2020

overview of similar discussion

Here an overview of similar discussions (where I constructed this answer from).

• How can I unload a DLL using ctypes in Python?
• ctypes unload dll
• Unload shared library inside ctypes loaded shared library
• forcing ctypes.cdll.LoadLibrary() to reload library from file

minimal reproducible example

This is for windows and linux, hence there are 2 scripts given for compilation. Tested under:

• Win 8.1, Python 3.8.3 (anaconda), ctypes 1.1.0, mingw-w64 x86_64-8.1.0-posix-seh-rt_v6-rev0
• Linux Fedora 32, Python 3.7.6 (anaconda), ctypes 1.1.0, g++ 10.2.1

cpp_code.cpp

extern "C" int my_fct(int n)
{
    int factor = 10;
    return factor * n;
}

compile-linux.sh

#!/bin/bash
g++ cpp_code.cpp -shared -o myso.so

compile-windows.cmd

set gpp="C:\Program Files\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\bin\g++.exe"
%gpp% cpp_code.cpp -shared -o mydll.dll
PAUSE

Python code

from sys import platform
import ctypes


if platform == "linux" or platform == "linux2":
    # https://stackoverflow.com/a/50986803/7128154
    # https://stackoverflow.com/a/52223168/7128154

    dlclose_func = ctypes.cdll.LoadLibrary('').dlclose
    dlclose_func.argtypes = [ctypes.c_void_p]

    fn_lib = './myso.so'
    ctypes_lib = ctypes.cdll.LoadLibrary(fn_lib)
    handle = ctypes_lib._handle

    valIn = 42
    valOut = ctypes_lib.my_fct(valIn)
    print(valIn, valOut)

    del ctypes_lib
    dlclose_func(handle)

elif platform == "win32": # Windows
    # https://stackoverflow.com/a/13129176/7128154
    # https://stackoverflow.com/questions/359498/how-can-i-unload-a-dll-using-ctypes-in-python

    lib = ctypes.WinDLL('./mydll.dll')
    libHandle = lib._handle

    # do stuff with lib in the usual way
    valIn = 42
    valOut = lib.my_fct(valIn)
    print(valIn, valOut)

    del lib
    ctypes.windll.kernel32.FreeLibrary(libHandle)

A more general solution (object oriented for shared libraries with dependencies)

If a shared library has dependencies, this does not necessarily work anymore (but it can - depends on the dependency ^^). I did not investigate the very details, but it looks like the mechanism is the following: library and dependency are loaded. As the dependency is not unloaded, the library can not get unloaded.

I found, that if I include OpenCv (Version 4.2) into my shared library, this messes up the unloading procedure. The following example was only tested on the linux system:

code.cpp

#include <opencv2/core/core.hpp>
#include <iostream> 


extern "C" int my_fct(int n)
{
    cv::Mat1b mat = cv::Mat1b(10,8,(unsigned char) 1 );  // change 1 to test unloading
    
    return mat(0,1) * n;
}

Compile with g++ code.cpp -shared -fPIC -Wall -std=c++17 -I/usr/include/opencv4 -lopencv_core -o so_opencv.so

Python code

from sys import platform
import ctypes


class CtypesLib:

    def __init__(self, fp_lib, dependencies=[]):
        self._dependencies = [CtypesLib(fp_dep) for fp_dep in dependencies]

        if platform == "linux" or platform == "linux2":  # Linux
            self._dlclose_func = ctypes.cdll.LoadLibrary('').dlclose
            self._dlclose_func.argtypes = [ctypes.c_void_p]
            self._ctypes_lib = ctypes.cdll.LoadLibrary(fp_lib)
        elif platform == "win32":  # Windows
            self._ctypes_lib = ctypes.WinDLL(fp_lib)

        self._handle = self._ctypes_lib._handle

    def __getattr__(self, attr):
        return self._ctypes_lib.__getattr__(attr)

    def __del__(self):
        for dep in self._dependencies:
            del dep

        del self._ctypes_lib

        if platform == "linux" or platform == "linux2":  # Linux
            self._dlclose_func(self._handle)
        elif platform == "win32":  # Windows
            ctypes.windll.kernel32.FreeLibrary(self._handle)


fp_lib = './so_opencv.so'

ctypes_lib = CtypesLib(fp_lib, ['/usr/lib64/libopencv_core.so'])

valIn = 1
ctypes_lib.my_fct.argtypes = [ctypes.c_int]
ctypes_lib.my_fct.restype = ctypes.c_int
valOut = ctypes_lib.my_fct(valIn)
print(valIn, valOut)

del ctypes_lib

Let me know, when there are any issues with the code examples or the explanation given so far. Also if you know a better way! It would be great, if we could settle the issue once and for all.

Answer 4

For total cross-compatibility: I maintain a list of various dlclose() equivalents for each platform and which library to get them from. It's a bit of a long list but feel free to just copy/paste it.

import sys
import ctypes
import platform

OS = platform.system()

if OS == "Windows":  # pragma: Windows
    dll_close = ctypes.windll.kernel32.FreeLibrary

elif OS == "Darwin":
    try:
        try:
            # macOS 11 (Big Sur). Possibly also later macOS 10s.
            stdlib = ctypes.CDLL("libc.dylib")
        except OSError:
            stdlib = ctypes.CDLL("libSystem")
    except OSError:
        # Older macOSs. Not only is the name inconsistent but it's
        # not even in PATH.
        stdlib = ctypes.CDLL("/usr/lib/system/libsystem_c.dylib")
    dll_close = stdlib.dlclose

elif OS == "Linux":
    try:
        stdlib = ctypes.CDLL("")
    except OSError:
        # Alpine Linux.
        stdlib = ctypes.CDLL("libc.so")
    dll_close = stdlib.dlclose

elif sys.platform == "msys":
    # msys can also use `ctypes.CDLL("kernel32.dll").FreeLibrary()`. Not sure
    # if or what the difference is.
    stdlib = ctypes.CDLL("msys-2.0.dll")
    dll_close = stdlib.dlclose

elif sys.platform == "cygwin":
    stdlib = ctypes.CDLL("cygwin1.dll")
    dll_close = stdlib.dlclose

elif OS == "FreeBSD":
    # FreeBSD uses `/usr/lib/libc.so.7` where `7` is another version number.
    # It is not in PATH but using its name instead of its path is somehow the
    # only way to open it. The name must include the .so.7 suffix.
    stdlib = ctypes.CDLL("libc.so.7")
    dll_close = stdlib.close

else:
    raise NotImplementedError("Unknown platform.")

dll_close.argtypes = [ctypes.c_void_p]

You can then use dll_close(dll._handle) to unload a library dll = ctypes.CDLL("your-library").

This list is taken from this file. I will update the master branch every time I encounter a new platform.

Answer 5

Piotr's answer helped me, but I did run into one issue on 64-bit Windows:

Traceback (most recent call last):
...
ctypes.ArgumentError: argument 1: <class 'OverflowError'>: int too long to convert

Adjusting the argument type of the FreeLibrary call as suggested in this answer solved this for me.

Thus we arrive at the following complete solution:

import ctypes, ctypes.windll

def free_library(handle):
    kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
    kernel32.FreeLibrary.argtypes = [ctypes.wintypes.HMODULE]
    kernel32.FreeLibrary(handle)

Usage:

lib = ctypes.CDLL("foobar.dll")
free_library(lib._handle)