Expose a vector as a memoryview using SWIG

Question

I have a header file like:

#include <vector>

inline std::vector<uint8_t>& vec() {
  static std::vector<uint8_t> v { 'a', 'b', 'c', 'd' };
  return v;
}

inline const std::vector<uint8_t>& cvec() {
  return vec();
}

I can wrap it in SWIG using std_vector.i and pyabc.i but that is quite inefficient (there's a jump between C++ and Python code for every access) and given that these are literally just a bunch of bytes I ought to be able to wrap them with Python's memoryview interface.

How can I expose my std::vector<uint8_t> as a Python memoryview?

Answer 1

Exposing it as a memoryview requires creating a Py_buffer first. In Python 3.3+ there is a convenient helper function, PyMemoryView_FromMemory that does a lot of the work for us. In earlier versions though we'll need to take a few extra steps, so our basic out typemap looks like:

%typemap(out) std::vector<uint8_t>&, const std::vector<uint8_t>& {
  Py_buffer *buf=(Py_buffer*)malloc(sizeof *buf);
  const bool ro = info<$1_type>::is_readonly();
  if (PyBuffer_FillInfo(buf, NULL,  &((*$1)[0]), (*$1).size(), ro, PyBUF_ND)) {
    // error, handle
  }
  $result = PyMemoryView_FromBuffer(buf);
}

Here we're basically allocating some memory for the Py_buffer. This just contains the details of the buffer internally for Python. The memory we allocate will be owned by the memoryview object once it's created. Unfortunately since it's going to be released with a call to free() we need to allocate it with malloc(), even though it's C++ code.

Besides the Py_buffer and an optional Py_Object PyBuffer_FillInfo takes a void* (the buffer itself), the size of the buffer, a boolean indicating if it's writeable and a flag. In this case our flag simply indicates that we have provided C-style contiguous memory for the buffer.

For deciding if it is readonly or not we used SWIG's built in $n_type variable and a helper (which could be a C++11 type trait if we wanted).

To complete our SWIG interface we need to provide that helper and include the header file, so the whole thing becomes:

%module test

%{
#include "test.hh" 

namespace {
  template <typename T>
  struct info {
    static bool is_readonly() {
      return false;
    }
  };

  template <typename T>
  struct info<const T&> {
    static bool is_readonly() {
      return true;
    }
  };
}
%}

%typemap(out) std::vector<uint8_t>&, const std::vector<uint8_t>& {
  Py_buffer *buf=(Py_buffer*)malloc(sizeof *buf);
  const bool ro = info<$1_type>::is_readonly();
  if (PyBuffer_FillInfo(buf, NULL,  &((*$1)[0]), (*$1).size(), ro, PyBUF_ND)) {
    // error, handle
  }
  $result = PyMemoryView_FromBuffer(buf);
}

%include "test.hh"

We can then test it with:

import test

print test.vec()
print len(test.vec())
print test.vec()[0]
print test.vec().readonly
test.vec()[0]='z'
print test.vec()[0]

print "This should fail:"
test.cvec()[0] = 0

Which worked as expected, tested using Python 2.7.

Compared to just wrapping it using std_vector.i this approach does have some drawbacks. The biggest being that we can't resize the vector, or convert it back to a vector later trivially. We could work around that, at least partially by creating a SWIG proxy for the vector like normal and using the second parameter of PyBuffer_FillInfo to store it internally. (This would also be needed if we had to manage the ownership of the vector for instance).