How do I make a struct for FFI that contains a nullable function pointer?

Question

I have an existing C program that loads shared library plugins. The main C program interacts with those plugins through a C struct containing integers, strings, function pointers, etc. How can I create such a plugin from Rust?

Note that the (real) C program cannot be changed, nor can the API be changed, those are fixed, existing things, so this is not a question about "how best to support plugins in Rust", it's how can Rust make *.so files which interoperate with an existing C program.

Here's a simplified example of a C program + C plugin:

/* gcc -g -Wall test.c -o test -ldl
   ./test ./test-api.so
 */

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <inttypes.h>
#include <dlfcn.h>

struct api {
  uint64_t i64;
  int i;
  const char *name;                /* can be NULL */
  void (*load) (void);             /* must not be NULL */
  void (*hello) (const char *str); /* can be NULL */
};

int
main (int argc, char *argv[])
{
  void *dl = dlopen (argv[1], RTLD_NOW);
  if (!dl) { fprintf (stderr, "%s: %s\n", argv[1], dlerror ()); exit (1); }
  struct api *(*get_api) (void) = dlsym (dl, "get_api");
  printf ("calling get_api ...\n");
  struct api *api = get_api ();
  printf ("api->i64 = %" PRIi64 "\n", api->i64);
  printf ("api->i = %d\n", api->i);
  if (api->name)
    printf ("api->name = %s\n", api->name);
  printf ("calling api->load ...\n");
  api->load ();
  if (api->hello) {
    printf ("calling api->hello ...\n");
    api->hello ("world");
  }
  printf ("exiting\n");
  exit (0);
}

/* gcc -g -shared -fPIC -Wall test-api.c -o test-api.so */

#include <stdio.h>
#include <stdint.h>

static void
load (void)
{
  printf ("this is the load function in the plugin\n");
}

static void
hello (const char *str)
{
  printf ("hello %s\n", str);
}

static struct api {
  uint64_t i64;
  int i;
  const char *name;
  void (*load) (void);
  void (*hello) (const char *str);
} api = {
  1042,
  42,
  "this is the plugin",
  load,
  hello,
};

struct api *
get_api (void)
{
  return &api;
}

Here's what I wrote in Rust to try to get a plugin, but it doesn't compile:

extern crate libc;

use libc::*;
use std::ffi::*;
use std::ptr;
use std::os::raw::c_int;

#[repr(C)]
pub struct api {
    i64: uint64_t,
    i: c_int,

    name: *const c_char,

    load: extern fn (),
    hello: extern fn (), // XXX
}

extern fn hello_load () {
    println! ("hello this is the load method");
}

#[no_mangle]
pub extern fn get_api () -> *const api {
    println! ("hello from the plugin");

    let api = Box::new (api {
        i64: 4201,
        i: 24,
        name: CString::new("hello").unwrap().into_raw(), // XXX memory leak?
        load: hello_load,
        hello: std::ptr::null_mut,
    });

    return Box::into_raw(api); // XXX memory leak?
}

This is compiled using Cargo.toml containing:

[package]
name = "embed"
version = "0.1.0"

[dependencies]
libc = "0.2"

[lib]
name = "embed"
crate-type = ["cdylib"]

The error is:

error[E0308]: mismatched types
  --> src/lib.rs:32:16
   |
32 |         hello: std::ptr::null_mut,
   |                ^^^^^^^^^^^^^^^^^^ expected "C" fn, found "Rust" fn
   |
   = note: expected type `extern "C" fn()`
              found type `fn() -> *mut _ {std::ptr::null_mut::<_>}`

error: aborting due to previous error

I didn't get to try loading the module but when I tried this before with the real program the fields were all wrong indicating something much more fundamental was wrong.

Answer 1

tl;dr Use Option to represent nullable function pointers and None for null.

The error message is confusing, first, because std::ptr::null_mut isn't a pointer; it's a generic function that returns a pointer, and you haven't called it. So Rust is seeing you pass a function that has the wrong signature and calling convention, and complaining about that.

But once you fix that, you'll get this error instead:

error[E0308]: mismatched types
  --> src/lib.rs:29:16
   |
29 |         hello: std::ptr::null_mut(),
   |                ^^^^^^^^^^^^^^^^^^^^ expected fn pointer, found *-ptr
   |
   = note: expected type `extern "C" fn()`
              found type `*mut _`

Function pointers and object pointers are not compatible (this is also the case in C), so you can't cast between them. null_mut returns an object pointer, so you need to find another way to create a null function pointer.

Function pointers (values of type fn(...) -> _) have another interesting property: unlike raw pointers (*const _ and *mut _), they can't be null. You don't need an unsafe block to call a function via pointer, and so creating a null function pointer is unsafe, like creating a null reference.

How do you make something nullable? Wrap it in Option:

#[repr(C)]
pub struct api {
    // ...
    load: Option<extern fn ()>,
    hello: Option<extern fn ()>, // assuming hello can also be null
}

And populate it with Some(function) or None:

let api = Box::new (api {
    // ...
    load: Some(hello_load),
    hello: None,
});

It's not usually a good idea to use enums, including Option, in a repr(C) struct, because C doesn't have an enum equivalent and so you don't know what you're going to get on the other side. But in the case of Option<T> where T is something non-nullable, None is represented by the null value, so it should be okay.

The use of Option to represent a nullable function pointer for FFI is documented in the Unsafe Code Guidelines:

null values are not supported by the Rust function pointer types -- just like references, the expectation is that you use Option to create nullable pointers. Option<fn(Args...) -> Ret> will have the exact same ABI as fn(Args...) -> Ret, but additionally allows null pointer values.