process video stream from memory buffer

Question

I need to parse a video stream (mpeg ts) from proprietary network protocol (which I already know how to do) and then I would like to use OpenCV to process the video stream into frames. I know how to use cv::VideoCapture from a file or from a standard URL, but I would like to setup OpenCV to read from a buffer(s) in memory where I can store the video stream data until it is needed. Is there a way to setup a call back method (or any other interfrace) so that I can still use the cv::VideoCapture object? Is there a better way to accomplish processing the video with out writing it out to a file and then re-reading it. I would also entertain using FFMPEG directly if that is a better choice. I think I can convert AVFrames to Mat if needed.

Answer 1

I had a similar need recently. I was looking for a way in OpenCV to play a video that was already in memory, but without ever having to write the video file to disk. I found out that the FFMPEG interface already supports this through av_open_input_stream. There is just a little more prep work required compared to the av_open_input_file call used in OpenCV to open a file.

Between the following two websites I was able to piece together a working solution using the ffmpeg calls. Please refer to the information on these websites for more details:

http://ffmpeg.arrozcru.org/forum/viewtopic.php?f=8&t=1170

http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/

To get it working in OpenCV, I ended up adding a new function to the CvCapture_FFMPEG class:

virtual bool openBuffer( unsigned char* pBuffer, unsigned int bufLen );

I provided access to it through a new API call in the highgui DLL, similar to cvCreateFileCapture. The new openBuffer function is basically the same as the open( const char* _filename ) function with the following difference:

err = av_open_input_file(&ic, _filename, NULL, 0, NULL);

is replaced by:

ic = avformat_alloc_context();
ic->pb = avio_alloc_context(pBuffer, bufLen, 0, pBuffer, read_buffer, NULL, NULL);

if(!ic->pb) {
    // handle error
}

// Need to probe buffer for input format unless you already know it
AVProbeData probe_data;
probe_data.buf_size = (bufLen < 4096) ? bufLen : 4096;
probe_data.filename = "stream";
probe_data.buf = (unsigned char *) malloc(probe_data.buf_size);
memcpy(probe_data.buf, pBuffer, probe_data.buf_size);

AVInputFormat *pAVInputFormat = av_probe_input_format(&probe_data, 1);

if(!pAVInputFormat)
    pAVInputFormat = av_probe_input_format(&probe_data, 0);

// cleanup
free(probe_data.buf);
probe_data.buf = NULL;

if(!pAVInputFormat) {
    // handle error
}

pAVInputFormat->flags |= AVFMT_NOFILE;

err = av_open_input_stream(&ic , ic->pb, "stream", pAVInputFormat, NULL);

Also, make sure to call av_close_input_stream in the CvCapture_FFMPEG::close() function instead of av_close_input_file in this situation.

Now the read_buffer callback function that is passed in to avio_alloc_context I defined as:

static int read_buffer(void *opaque, uint8_t *buf, int buf_size)
{
    // This function must fill the buffer with data and return number of bytes copied.
    // opaque is the pointer to private_data in the call to avio_alloc_context (4th param)
    
    memcpy(buf, opaque, buf_size);
    return buf_size;
}

This solution assumes the entire video is contained in a memory buffer and would probably have to be tweaked to work with streaming data.

So that's it! Btw, I'm using OpenCV version 2.1 so YMMV.

Answer 2

Code to do similar to the above, for opencv 4.2.0 is on: https://github.com/jcdutton/opencv Branch: 4.2.0-jcd1

Load the entire file into RAM pointed to by buffer, of size buffer_size. Sample code:

VideoCapture d_reader1;
d_reader1.open_buffer(buffer, buffer_size);
d_reader1.read(input1);

The above code reads the first frame of video.