How to write a video file with OpenCV?

Question

I am trying to write a program to analyze emotional expressions like tears. As part of my tracker I am using OpenCV to record sample videos. Particularly, I am not certain about how to correctly choose FPS (10FPS seems like it ought to work). I am also not sure which Codec I should use on OS X, I have tried all possible CV_FOURCC from here as well but returned the following error:

Stream #0.0: Video: rawvideo, yuv420p, 640x480, q=2-31, 19660 kb/s, 90k tbn, 10 tbc
Assertion failed: (image->imageSize == avpicture_get_size( (PixelFormat)input_pix_fmt, image->width, image->height )), function writeFrame, file /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_graphics_opencv/work/OpenCV-2.2.0/modules/highgui/src/cap_ffmpeg.cpp, line 1085.

Do you all have some working code with cvWriteFrame? Thanks for taking time to look at my problem!

For those interested the entire program is:

#include <cv.h>
#include <cxcore.h>
#include <highgui.h>

int main (int argc, const char * argv[]) 
{
    CvCapture *capture;
    IplImage  *img;
    int       key = 0;
    CvVideoWriter *writer;

    // initialize camera
    capture = cvCaptureFromCAM( 0 );
    // capture = cvCaptureFromAVI("AVIFile");

    // always check
    assert( capture );

    // create a window
    cvNamedWindow( "video", 1 );

    int color = 1; // 0 for black and white

    // get the frame size
    CvSize size = cvSize((int)cvGetCaptureProperty( capture, CV_CAP_PROP_FRAME_WIDTH),(int)cvGetCaptureProperty( capture, CV_CAP_PROP_FRAME_HEIGHT));

    writer = cvCreateVideoWriter(argv[1], -1 , 10 , size, color);

    while( key != 'q' ) {
        // get a frame
        img = cvQueryFrame( capture );

        // always check
        if( !img ) break;

        cvWriteFrame( writer, img );        
        cvShowImage("video", img );

        // quit if user press 'q'
        key = cvWaitKey( 5 );
    }

    // free memory
    cvReleaseVideoWriter( &writer );
    cvReleaseCapture( &capture );
    cvDestroyWindow( "video" );

    return 0;   
}

Answer 1

Have a look here for the recommended codecs to use.

Basically:

• You can use -1 instead of CV_FOURCC(...) only if you're on Windows (see the manual). Since you're on OSX, you can't use it.
• Use CV_FOURCC('I', 'Y', 'U', 'V') to encode using yuv420p into an uncompressed AVI container -- this is the recommended approach
• If you want compressed video, then you can use tools like ffmpeg to compress the AVI output by OpenCV

The reason OpenCV doesn't recommend using compressor codecs is probably because each codec has a large number of codec-specific options, and dealing with all of them through the simple OpenCV video writer interface would be impossible.

EDIT

I have some bad news:

Unfortunately, we still have no working (i.e. native) video writer for Mac OS X. If you need video writing, you should currently configure for ffmpeg or xine and disable QuickTime.

Sounds like you may need to reconfigure and recompile your library.

EDIT 2

Alternatively, you could just write each frame to a JPEG file, and then stitch those JPEG files into a movie using ffmpeg (that should work on OS/X). This is essentially what MJPEG is, anyway. You'll probably get better compression rates with codecs that exploit temporal redundancy (e.g. MPEG4, H.264, etc), though.

Answer 2

The assertion suggests that there is a size mismatch between the image being sent to the video writer, and the size specified when initializing the video writer. I would suggest adding some debug statements as follows:

cout << "Video Size: " << size.width << "," << size.height << endl;
cout << "Image Size: " << img.cols << "," << img.rows << endl;

That should help you find out if the sizes are different.

With regard to the fourcc question, it looks like you are using cap_ffmpeg instead of cap_qtkit. You may want to try recompiling with ffmpeg off, since the qtkit version of capture works pretty well on OS X. Rather than using macports, you can directly download opencv2.2 from sourceforge and use ccmake to configure opencv not to use ffmpeg, in that case qtkit will be the default.

If you do use qtkit, then you need to use the fourcc('j','p','e','g');

The frame rate should match the frame rate of the input. You can use the properties to get the frame rate similar to the way you got the frame width and height, however since you are capturing from the camera, you will probably have to test the capture rate by using a timer in your code and estimating from that.

Here is an example based on starter_video.cpp that will work with qtkit:

/*
* starter_video_write.cpp
*
*  Created on: Nov 23, 2010
*      Author: Ethan Rublee
*  Modified on: Aug 3, 2011 by David Cooper
*
* A starter sample for using opencv, get a video stream and display the images
* easy as CV_PI right?
* edited to add writing capability for OS X using qtkit.
*/
#include "opencv2/highgui/highgui.hpp"
#include <iostream>
#include <vector>
#include <stdio.h>

using namespace cv;
using namespace std;



//hide the local functions in an anon namespace
namespace {
    void help(char** av) {
        cout << "\nThis program justs gets you started reading images from video\n"
            "Usage:\n./" << av[0] << " <video device number>\n"
            << "q,Q,esc -- quit\n"
            << "space   -- save frame\n\n"
            << "\tThis is a starter sample, to get you up and going in a copy pasta fashion\n"
            << "\tThe program captures frames from a camera connected to your computer.\n"
            << "\tTo find the video device number, try ls /dev/video* \n"
            << "\tYou may also pass a video file, like my_vide.avi instead of a device number"
            << endl;
    }

    int process(VideoCapture& capture) {
        int n = 0;
        char filename[200];
        string window_name = "video | q or esc to quit";
        cout << "press space to save a picture. q or esc to quit" << endl;
        namedWindow(window_name, CV_WINDOW_KEEPRATIO); //resizable window;
        Mat frame;
    double frameRate = 15.0;
    capture >> frame;
    VideoWriter* writer = NULL;
    // FIXME: need to change this to an absolute path that exists on this system
    string outputFile = "/tmp/test.mov";
    writer = new VideoWriter(outputFile,
                 CV_FOURCC('j','p','e','g'),
                 frameRate,Size(frame.cols,frame.rows));


        for (;;) {
            capture >> frame;
            if (frame.empty())
                continue;
            imshow(window_name, frame);
        writer->write(frame);
            char key = (char)waitKey(5); //delay N millis, usually long enough to display and capture input
            switch (key) {
        case 'q':
        case 'Q':
        case 27: //escape key
          if(writer != NULL) {
        delete writer;
          }
          return 0;
        case ' ': //Save an image
          sprintf(filename,"filename%.3d.jpg",n++);
          imwrite(filename,frame);
          cout << "Saved " << filename << endl;
          break;
        default:
          break;
            }
        }
        return 0;
    }

}

int main(int ac, char** av) {

    if (ac != 2) {
        help(av);
        return 1;
    }
    std::string arg = av[1];
    VideoCapture capture(arg); //try to open string, this will attempt to open it as a video file
    if (!capture.isOpened()) //if this fails, try to open as a video camera, through the use of an integer param
        capture.open(atoi(arg.c_str()));
    if (!capture.isOpened()) {
        cerr << "Failed to open a video device or video file!\n" << endl;
        help(av);
        return 1;
    }
    return process(capture);
}