Wrapping C++ Function That Has OpenCV Parameters with Cython

Question

I have the following class that I have written in C++

#include "segmenter_interface.h"
#include "video_marker.cpp"
#include <opencv2/imgcodecs.hpp>
#include <opencv2/videoio/videoio.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/opencv.hpp>
#include <vector>

class UserDivide : public SegmenterInterface{
  public: 
    virtual void SegmentVideo(cv::VideoCapture *vc, 
        std::vector<Segment> *indices); 
}

The implementation details are unimportant. Ideally, I would like to expose this class to python using Cython. The object VideoCapture can already be instantiated by python because OpenCV wraps all of its modules already. From what I have read, vector is already part of Cython since it supports most of the C++ standard libraries.

Currently, I have written this much .pyx:

cdef extern from "<vector>" namespace "std":
    cdef cppclass vector [T]: 
        pass

cdef extern from "<opencv2/videoio/videoio.hpp>" namespace "cv": 
    cdef cppclass VideoCapture: 
        pass # Don't care, just need a pointer

cdef extern from "segmenter_interface.h":
    cdef struct Segment: 
        pass # I will need this eventually...

cdef extern from "user_divide.h":
    cdef cppclass UserDivide: 
        UserDivide()
        void SegmentVideo(VideoCapture *vc, vector[Segment] *indices)  

cdef class PyUserDivide:
    cdef UserDivide *thisptr # hold a C++ instance
    def __cinit__(self): 
        self.thisptr = new UserDivide()
    def __dealloc__(self): 
        del self.thisptr
    def SegmentVideo(self, VideoCapture *vc, vector[Segment] *indices): 
        return self.thisptr.SegmentVideo(vc, indices)

The problem is the parameters in SegmentVideo--the cython compiler complains about converting them.

My question is this: How do I properly wrap object pointers in Cython, especially if it's not a standard data type or struct, but a class? Am I even taking the correct approach?

My goal is to do the following in Python

import cv2
cap = cv2.VideoCapture("my_video.mp4")
segments = []
ud = UserDivide()
ud.SegmentVideo(cap, segments)
# Do stuff with segments

The error message is the following:

Error compiling Cython file:
------------------------------------------------------------
...
    cdef UserDivide *thisptr # hold a C++ instance
    def __cinit__(self): 
        self.thisptr = new UserDivide()
    def __dealloc__(self): 
        del self.thisptr
    def SegmentVideo(self, VideoCapture *vc, vector[Segment] *indices): 
                          ^
------------------------------------------------------------

pyuser_divide.pyx:22:27: Cannot convert Python object argument to type 'VideoCapture *'

Error compiling Cython file:
------------------------------------------------------------
...
    cdef UserDivide *thisptr # hold a C++ instance
    def __cinit__(self): 
        self.thisptr = new UserDivide()
    def __dealloc__(self): 
        del self.thisptr
    def SegmentVideo(self, VideoCapture *vc, vector[Segment] *indices): 
                                            ^
------------------------------------------------------------

pyuser_divide.pyx:22:45: Cannot convert Python object argument to type 'vector[Segment] *'

Answer 1

The is unfortunately a partial answer: I know the problem but I don't know the solution.

Your second problem is easily solved. The issue is that it expects a pointer to a c++ vector, but it gets a python list. Cython can automatically convert a list to a vector and back, but gets lost at the pointer.

 # put this at the of your file top, and remove
 # cdef extern from "<vector>" namespace "std": (etc)
 #
 # Cython has automatically defined these!
 from libcpp.vector cimport vector

 # ... other code goes here ...

 # accept a list
 def SegmentVideo(self, VideoCapture *vc, list indices):
    cdef vector[Segment] indices_vector = indices # autoconversion happens
    output = self.thisptr.SegmentVideo(vc, &indices_vector)
    # are you expecting the vector to be changed, and do you want to keep the changes?
    # if so then do
    indices[:] = indices_vector
    # if you don't care about changes then don't bother
    return output

Your first problem (Cannot convert Python object argument to type 'VideoCapture *') is the one I can't fix. Essentially, the cv2 module generates python objects (which are presumably wrappers around C++ pointers defined in OpenCV), and there isn't an obvious way to generate a C++ pointer from those python objects - you'd have to figure out where it's stored.

A quick poke around which I won't repeat in full suggests the VideoCapture python objects are 32 bytes on my system (an empty Python object is 16 bytes). However, neither of the contents of the 2 extra 8-byte spaces "look like pointers" to me.

Most of the OpenCV python wrapper looks to be self generating from the C++ code (at least as far as I can tell), meaning it isn't particularly easy to work out how it's wrapped. A possible approach that might be useful would be to try running UserDivide through their code-generator and see if you can generate python bindings for it that way....