Python ctypes: pass argument by reference error

Question

I have a C++ function that I want you call in Python 2.7.12, looking like this:

extern "C" {
    double* myfunction(double* &y, double* &z, int &n_y, int &n_z, int a, int b)
    {
        vector<double> _x;
        vector<double> _y;
        vector<double> _z;

        // Call some external C++ function
        cpp_function(_x, _y, _z, a, b);

        // Convert vectors back to arrays
        double* x = &_x[0]; // or x = _x.data();
        y = &_y[0];
        z = &_z[0];
        n_y = static_cast<int>(_y.size());
        n_z = static_cast<int>(_z.size());
        return x;
    }
}

Basically this function takes as input two integers a,b (plus some other data that I omitted for clarity purpose) and do some calculations before putting the results into two arrays y, z and their respective sizes into n_y, n_z, and returning an array x of size a*b.

After building this function to a shared library myfunction.so, I call it in Python as follows:

from ctypes import *

libc = CDLL('myfunction.so')
myfunction = libc.myfunction

myfunction.restype = POINTER(c_double)
myfunction.argtypes = [POINTER(c_double), POINTER(c_double),
                       c_int, c_int,
                       c_int, c_int]

y = POINTER(c_double)()
z = POINTER(c_double)()
n_y = c_int()
n_z = c_int()

a = 18
b = 18
x = myfunction(byref(y), byref(z),
               byref(n_y), byref(n_z),
               c_int(a), c_int(b))

Running this script I obtained an error:

ctypes.ArgumentError: argument 3: : wrong type

So the c_int type of n_y is not correct. What should I put instead?

Thank you very much for your help!

---

UPDATE

Following the suggestion by @GiacomoAlzetta and @CristiFati, I have changed my code to use pointers instead of pass by reference, as follows.

(y and z are similar so let me omit z)

extern "C" {
    double* myfunction(double** y, int* n_y, int a, int b)
    {
        vector<double> _x;
        vector<double> _y;

        // Call some external C++ function
        cpp_function(_x, _y, a, b);

        // Convert vectors back to arrays
        double* x = &_x[0]; // or x = _x.data();
        *y = &_y[0];
        *n_y = static_cast<int>(_y.size());
        return x;
    }
}

Now in C++, I call the above function as follows:

double* y;
int n_y;
int a = 18;
int b = 18;
double* x = myfunction(&y, &n_y, a, b);

which works. And in Python:

from ctypes import *

libc = CDLL('myfunction.so')
myfunction = libc.myfunction

myfunction.restype = POINTER(c_double)
myfunction.argtypes = [POINTER(POINTER(c_double)), POINTER(c_int),
                       c_int, c_int]

y = POINTER(POINTER(c_double))()
n_y = POINTER(c_int)()

a = 18
b = 18
x = myfunction(y, n_y, c_int(a), c_int(b))

which produced a Segmentation fault error, which happened at the line

*y = &_y[0];

Thank you for your help!

Answer 1

You're almost there.
In the meantime, stay close to [Python 3.Docs]: ctypes - A foreign function library for Python.

Remember that you should handle pointer arguments (actually it applies to all of them, but for non pointer ones things are straightforward) the same way, no matter where you are.

In other words, what you do in C (instantiate a variable and pass its pointer to the function), you should also do in Python (instead of instantiate the variable pointer and pass it to the function).

Translated into code, you should modify the way you initialize y, n_y, and the function (myfunction) call:

>>> from ctypes import *  # Anti-pattern. Don't ever use it
>>>
>>> y = POINTER(c_double)()
n_y = c_int()
a = 18
b = 18
x = myfunction(pointer(y), pointer(n_y), a, b)

Notes:

• What I stated in a comment (Undefined Behavior because vectors are living on the stack and will be destroyed when exiting the function) still stands. To fix it either:
  • Allocate the data on heap (malloc / new) before returning it (when done with it, you'll also need to deallocate it (free / delete), to avoid memory leaks)
  • Make them static

Some remotely connected examples:

• [SO]: Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated (@CristiFati's answer)
• [SO]: python ctypes issue on different OSes (@CristiFati's answer)

Answer 2

You can use references, since references are just syntax for yet another level of pointer.

You vectors are local variables and are freed when your function returns, so you need to keep the memory around.

Here's your C++ code reworked to keep the memory around. I just created some local variables with some data since your example wasn't complete:

#define API __declspec(dllexport)  // Windows-specific export
#include <cstdlib>
#include <vector>

using namespace std;

extern "C" {
    API double* myfunction(double* &y, double* &z, int &n_x, int &n_y, int &n_z)
    {
        vector<double> _x {1.1,2.2,3.3};
        vector<double> _y {4.4,5.5};
        vector<double> _z {6.6,7.7,8.8,9.9};

        // Allocate some arrays to store the vectors.
        double* x = new double[_x.size()];
        y = new double[_y.size()];
        z = new double[_z.size()];
        memcpy(x,_x.data(),_x.size() * sizeof(double));
        memcpy(y,_y.data(),_y.size() * sizeof(double));
        memcpy(z,_z.data(),_z.size() * sizeof(double));
        n_x = static_cast<int>(_x.size());
        n_y = static_cast<int>(_y.size());
        n_z = static_cast<int>(_z.size());
        return x;
    }

    // A function to free up the memory.
    API void myfree(double* x, double* y, double* z)
    {
        delete [] x;
        delete [] y;
        delete [] z;
    }
}

Python:

from ctypes import *

dll = CDLL('test')
dll.myfunction.argtypes = (POINTER(POINTER(c_double)),
                           POINTER(POINTER(c_double)),
                           POINTER(c_int),
                           POINTER(c_int),
                           POINTER(c_int))
dll.myfunction.restype = POINTER(c_double)

dll.myfree.argtypes = POINTER(c_double),POINTER(c_double),POINTER(c_double)
dll.myfree.restype = None

# Helper function to allocate storage for return arrays
def myfunction():
    y = POINTER(c_double)() # create an instance of a C double*
    z = POINTER(c_double)()
    n_x = c_int()           # and instances of C int
    n_y = c_int()
    n_z = c_int()

    # Pass them all by reference so new values can be returned
    x = dll.myfunction(byref(y),byref(z),byref(n_x),byref(n_y),byref(n_z))

    # Copies the data into Python lists
    a = x[:n_x.value]
    b = y[:n_y.value]
    c = z[:n_z.value]

    # Free the C arrays and return the Python lists.
    dll.myfree(x,y,z)
    return a,b,c

x,y,z = myfunction()
print(x,y,z)

Output:

[1.1, 2.2, 3.3] [4.4, 5.5] [6.6, 7.7, 8.8, 9.9]

Note there is a lot of copying going on. Look into numpy, which creates arrays in a format that can be directly accessed by C and has a built-in ctypes interface.