Pass Python list to embedded Rust function

Question

I am learning how to embed Rust functions in Python, and everything works fine if my inputs are ints, but not list.

If my lib.rs file is:

#[no_mangle]
pub extern fn my_func(x: i32, y: i32) -> i32 {
    return x + y;
}

I can use this as follows:

In [1]: from ctypes import cdll

In [2]: lib = cdll.LoadLibrary("/home/user/RustStuff/embed/target/release/libembed.so")

In [3]: lib.my_func(5,6)
Out[3]: 11

However if I change my lib.rs to:

#[no_mangle]
pub extern fn my_func(my_vec: Vec<i32>) -> i32 {
    let mut my_sum = 0;
    for i in my_vec {
        my_sum += i;
    }
    return my_sum;
}

I can no longer use it in Python (this compiled fine):

In [1]: from ctypes import cdll

In [2]: lib = cdll.LoadLibrary("/home/user/RustStuff/embed/target/release/libembed.so")

In [3]: lib.my_func([2,3,4])
---------------------------------------------------------------------------
ArgumentError                             Traceback (most recent call last)
<ipython-input-3-454ffc5ba9dd> in <module>()
----> 1 lib.my_func([2,3,4])

ArgumentError: argument 1: <type 'exceptions.TypeError'>: Don't know how to convert parameter 1

The reason, I though this could work is that Python's list and Rust's Vec are the both dynamic arrays, but apparently I am missing something here...

Why does my attempt not work? What should I do to fix it?

Answer 1

Don't do this:

#[no_mangle]
pub extern fn my_func(my_vec: Vec<i32>) -> i32 { ... }

You basically never want to accept or return an arbitrary Rust object in an extern function, only ones that are Repr. Instead, you should accept something that is representable by C. As 6502 says, the best idea for this particular case would be to accept a pointer and a length.

Rust's Vec is conceptually a pointer to data, a count, and a capacity. You are able to modify a Vec by adding or removing objects, which can cause reallocation to happen. This is doubly bad because it is likely that Python and Rust use different allocators that are not compatible with each other. Segfaults lie this way! You really want a slice.

Instead, do something like this on the Rust side:

extern crate libc;

use libc::{size_t,int32_t};
use std::slice;

#[no_mangle]
pub extern fn my_func(data: *const int32_t, length: size_t) -> int32_t {
    let nums = unsafe { slice::from_raw_parts(data, length as usize) };
    nums.iter().fold(0, |acc, i| acc + i)
}

Namely, you are using the guaranteed-to-match C types, and then converting the pointer and length to something Rust knows how to deal with.

I'm no Pythonista, but this cobbled-together code (with help from How do I convert a Python list into a C array by using ctypes?) seems to work with the Rust I have above:

import ctypes

lib = ctypes.cdll.LoadLibrary("./target/debug/libpython.dylib")
lib.my_func.argtypes = (ctypes.POINTER(ctypes.c_int32), ctypes.c_size_t)

list_to_sum = [1,2,3,4]
c_array = (ctypes.c_int32 * len(list_to_sum))(*list_to_sum)
print lib.my_func(c_array, len(list_to_sum))

Of course, you probably want to wrap that to make it nicer for the caller of your code.

Answer 2

ctypes is about C bindings and in C there's no such a thing as a dynamic array.

The closest object you can pass to a C function is a pointer to integer that however is not a dynamic array because

1. It doesn't carry the size information
2. You cannot grow or shrink the area, just access existing elements

A simple alternative to passing pointers (and being very careful about not getting past the size) you could use instead is a function-based API.

For example:

getNumberOfThings() -> number
getThing(index) -> thing

but the Python code would then become like

def func():
    n = getNumberOfThings()
    return [getThing(i) for i in range(n)]

The counterpart (passing a variable number of elements) would be

def func2(L):
    setNumberOfThings(len(L))
    for i, x in enumerate(L):
        setThing(i, x)