Passing a C++ std::Vector to numpy array in Python

Question

I am trying a pass a vector of doubles that I generate in my C++ code to a python numpy array. I am looking to do some downstream processing in Python and want to use some python facilities, once I populate the numpy array. One of the biggest things I want to do is to be able to plot things, and C++ is a bit clumsy when it comes to that. Also I want to be able to leverage Python's statistical power.

Though I am not very clear as to how to do it. I spent a lot of time going through the Python C API documentation. I came across a function PyArray_SimpleNewFromData that apparently can do the trick. I still am very unclear as far as the overall set up of the code is concerned. I am building certain very simple test cases to help me understand this process. I generated the following code as a standlone Empty project in Visual Studio express 2012. I call this file Project1

#include <Python.h>
#include "C:/Python27/Lib/site-packages/numpy/core/include/numpy/arrayobject.h"

PyObject * testCreatArray()
{
    float fArray[5] = {0,1,2,3,4};
    npy_intp m = 5;
    PyObject * c = PyArray_SimpleNewFromData(1,&m,PyArray_FLOAT,fArray);
    return c; 
}

My goal is to be able to read the PyObject in Python. I am stuck because I don't know how to reference this module in Python. In particular how do I import this Project from Python, I tried to do a import Project1, from the project path in python, but failed. Once I understand this base case, my goal is to figure out a way to pass the vector container that I compute in my main function to Python. I am not sure how to do that either.

Any experts who can help me with this, or maybe post a simple well contained example of some code that reads in and populates a numpy array from a simple c++ vector, I will be grateful. Many thanks in advance.

Answer 1

I'm not a cpp-hero ,but wanted to provide my solution with two template functions for 1D and 2D vectors. This is a one liner for usage l8ter and by templating 1D and 2D vectors, the compiler can take the correct version for your vectors shape. Throws a string in case of unregular shape in the case of 2D. The routine copies the data here, but one can easily modify it to take the adress of the first element of the input vector in order to make it just a "representation".

Usage looks like this:

// Random data
vector<float> some_vector_1D(3,1.f); // 3 entries set to 1
vector< vector<float> > some_vector_2D(3,vector<float>(3,1.f)); // 3 subvectors with 1

// Convert vectors to numpy arrays
PyObject* np_vec_1D = (PyObject*) vector_to_nparray(some_vector_1D);
PyObject* np_vec_2D = (PyObject*) vector_to_nparray(some_vector_2D);

You may also change the type of the numpy array by the optional arguments. The template functions are:

/** Convert a c++ 2D vector into a numpy array
 *
 * @param const vector< vector<T> >& vec : 2D vector data
 * @return PyArrayObject* array : converted numpy array
 *
 * Transforms an arbitrary 2D C++ vector into a numpy array. Throws in case of
 * unregular shape. The array may contain empty columns or something else, as
 * long as it's shape is square.
 *
 * Warning this routine makes a copy of the memory!
 */
template<typename T>
static PyArrayObject* vector_to_nparray(const vector< vector<T> >& vec, int type_num = PyArray_FLOAT){

   // rows not empty
   if( !vec.empty() ){

      // column not empty
      if( !vec[0].empty() ){

        size_t nRows = vec.size();
        size_t nCols = vec[0].size();
        npy_intp dims[2] = {nRows, nCols};
        PyArrayObject* vec_array = (PyArrayObject *) PyArray_SimpleNew(2, dims, type_num);

        T *vec_array_pointer = (T*) PyArray_DATA(vec_array);

        // copy vector line by line ... maybe could be done at one
        for (size_t iRow=0; iRow < vec.size(); ++iRow){

          if( vec[iRow].size() != nCols){
             Py_DECREF(vec_array); // delete
             throw(string("Can not convert vector<vector<T>> to np.array, since c++ matrix shape is not uniform."));
          }

          copy(vec[iRow].begin(),vec[iRow].end(),vec_array_pointer+iRow*nCols);
        }

        return vec_array;

     // Empty columns
     } else {
        npy_intp dims[2] = {vec.size(), 0};
        return (PyArrayObject*) PyArray_ZEROS(2, dims, PyArray_FLOAT, 0);
     }


   // no data at all
   } else {
      npy_intp dims[2] = {0, 0};
      return (PyArrayObject*) PyArray_ZEROS(2, dims, PyArray_FLOAT, 0);
   }

}


/** Convert a c++ vector into a numpy array
 *
 * @param const vector<T>& vec : 1D vector data
 * @return PyArrayObject* array : converted numpy array
 *
 * Transforms an arbitrary C++ vector into a numpy array. Throws in case of
 * unregular shape. The array may contain empty columns or something else, as
 * long as it's shape is square.
 *
 * Warning this routine makes a copy of the memory!
 */
template<typename T>
static PyArrayObject* vector_to_nparray(const vector<T>& vec, int type_num = PyArray_FLOAT){

   // rows not empty
   if( !vec.empty() ){

       size_t nRows = vec.size();
       npy_intp dims[1] = {nRows};

       PyArrayObject* vec_array = (PyArrayObject *) PyArray_SimpleNew(1, dims, type_num);
       T *vec_array_pointer = (T*) PyArray_DATA(vec_array);

       copy(vec.begin(),vec.end(),vec_array_pointer);
       return vec_array;

   // no data at all
   } else {
      npy_intp dims[1] = {0};
      return (PyArrayObject*) PyArray_ZEROS(1, dims, PyArray_FLOAT, 0);
   }

}

Answer 2

Since there is no answer to this that is actually helpful for people that might be looking for this sort of thing I figured I'd put an easy solution.

First you will need to create a python extension module in C++, this is easy enough to do and is all in the python c-api documentation so i'm not going to go into that.

Now to convert a c++ std::vector to a numpy array is extremely simple. You first need to import the numpy array header

#include <numpy/arrayobject.h>

and in your intialising function you need to import_array()

PyModINIT_FUNC
inittestFunction(void){
   (void) Py_InitModule("testFunction". testFunctionMethods);
   import_array();
}

now you can use the numpy array functions that are provided. The one that you will want for this is as the OP said a few years back PyArray_SimpleNewFromData, it's stupidly simple to use. All you need is an array of type npy_intp, this is the shape of the array to be created. make sure it is the same as your vector using testVector.size(), (and for multiple dimensions do testVector[0].size(), testVector[0][0].size() ect. vectors are guaranteed to be continuous in c++11 unless it's a bool).

//create testVector with data initialised to 0
std::vector<std::vector<uint16_t>> testVector;
testVector.resize(width, std::vector<uint16_t>(height, 0);
//create shape for numpy array
npy_intp dims[2] = {width, height}
//convert testVector to a numpy array
PyArrayObject* numpyArray = (PyArrayObject*)PyArray_SimpleNewFromData(2, dims, NPY_UINT16, (uint16_t*)testVector.data());

To go through the paramaters. First you need to cast it to a PyArrayObject, otherwise it will be a PyObject and when returned to python won't be a numpy array. The 2, is the number of dimensions in the array. dims, is the shape of the array. This has to be of type npy_intp NPY_UINT16 is the data type that the array will be in python. you then use testVector.data() to get the data of the array, cast this to either void* or a pointer of the same data type as your vector.

Hope this helps anyone else who may need this.

(Also if you don't need pure speed I would advise avoiding using the C-API, it causes quite a few problems and cython or swig are still probably your best choices. There is also c types which can be quite helpful.

Answer 3

I came across your post when trying to do something very similar. I was able to cobble together a solution, the entirety of which is on my Github. It makes two C++ vectors, converts them to Python tuples, passes them to Python, converts them to NumPy arrays, then plots them using Matplotlib.

Much of this code is from the Python Documentation.

Here are some of the important bits from the .cpp file :

 //Make some vectors containing the data
 static const double xarr[] = {1,2,3,4,5,6,7,8,9,10,11,12,13,14};
 std::vector<double> xvec (xarr, xarr + sizeof(xarr) / sizeof(xarr[0]) );
 static const double yarr[] = {0,0,1,1,0,0,2,2,0,0,1,1,0,0};
 std::vector<double> yvec (yarr, yarr + sizeof(yarr) / sizeof(yarr[0]) );

 //Transfer the C++ vector to a python tuple
 pXVec = PyTuple_New(xvec.size()); 
 for (i = 0; i < xvec.size(); ++i) {
      pValue = PyFloat_FromDouble(xvec[i]);
      if (!pValue) {
           Py_DECREF(pXVec);
           Py_DECREF(pModule);
           fprintf(stderr, "Cannot convert array value\n");
           return 1;
      }
      PyTuple_SetItem(pXVec, i, pValue);
 }

 //Transfer the other C++ vector to a python tuple
 pYVec = PyTuple_New(yvec.size()); 
 for (i = 0; i < yvec.size(); ++i) {
      pValue = PyFloat_FromDouble(yvec[i]);
      if (!pValue) {
           Py_DECREF(pYVec);
           Py_DECREF(pModule);
           fprintf(stderr, "Cannot convert array value\n");
           return 1;
      }
      PyTuple_SetItem(pYVec, i, pValue); //
 }

 //Set the argument tuple to contain the two input tuples
 PyTuple_SetItem(pArgTuple, 0, pXVec);
 PyTuple_SetItem(pArgTuple, 1, pYVec);

 //Call the python function
 pValue = PyObject_CallObject(pFunc, pArgTuple);

And the Python code:

def plotStdVectors(x, y):
    import numpy as np
    import matplotlib.pyplot as plt
    print "Printing from Python in plotStdVectors()"
    print x
    print y
    x = np.fromiter(x, dtype = np.float)
    y = np.fromiter(y, dtype = np.float)
    print x
    print y
    plt.plot(x, y)
    plt.show()
    return 0

Which results in the plot that I can't post here due to my reputation, but is posted on my blog post here.