Segmentation fault while avcodec_encode_video2

Question

I have some problems while trying to encode a AVFrame to a packet.

Before reading the whole code, the input stuff is working, I tested it. The output stuff is from an example here. I think there is the problem. But the segmentation fault occurs in the loop near the end.

Here is my reduced code:

void nmain() {

  // input stuff
  AVFormatContext *formatCtxIn=0;
  AVInputFormat *formatIn=0;
  AVCodecContext *codecCtxIn=0;
  AVCodec *codecIn;
  AVPacket *pktIn;

  av_register_all();
  avdevice_register_all();
  avcodec_register_all();


  formatIn = av_find_input_format("dshow");

  if(!formatIn)
    return;


  AVDictionary *avoption=0;
  av_dict_set(&avoption, "rtbufsize", "1000000000", NULL);

  if(avformat_open_input(&formatCtxIn, "video=Integrated Camera", formatIn, &avoption)!=0)
    return;

  if(avformat_find_stream_info(formatCtxIn, NULL)<0)
    return;

  codecCtxIn = formatCtxIn->streams[0]->codec;
  codecIn = avcodec_find_decoder(codecCtxIn->codec_id);

  if(avcodec_open2(codecCtxIn, codecIn, NULL)<0)
    return;


  // end input stuff  
//------------------------------------------------------------------------------
  // output stuff

  AVOutputFormat *formatOut=0;
  AVFormatContext *formatCtxOut=0;
  AVStream *streamOut=0;
  AVFrame *frame=0;
  AVCodec *codecOut=0;
  AVPacket *pktOut;

  const char *filename = "test.mpeg";

  formatOut = av_guess_format(NULL, filename, NULL);
  if(!formatOut)
    formatOut = av_guess_format("mpeg", NULL, NULL);
  if(!formatOut)
    return;

  formatCtxOut = avformat_alloc_context();
  if(!formatCtxOut)
    return;

  formatCtxOut->oformat = formatOut;

  sprintf(formatCtxOut->filename, "%s", filename);

  if(formatOut->video_codec != AV_CODEC_ID_NONE) {
    AVCodecContext *ctx;

    codecOut = avcodec_find_encoder(formatOut->video_codec);
    if(!codecOut)
      return;

    streamOut = avformat_new_stream(formatCtxOut, codecOut);
    if(!streamOut)
      return;

    ctx = streamOut->codec;

    ctx->bit_rate = 400000;
    ctx->width    = 352;
    ctx->height   = 288;
    ctx->time_base.den = 25;
    ctx->time_base.num = 1;
    ctx->gop_size      = 12;
    ctx->pix_fmt       = AV_PIX_FMT_YUV420P;

    if(ctx->codec_id == AV_CODEC_ID_MPEG2VIDEO)
      ctx->max_b_frames = 2;
    if(ctx->codec_id == AV_CODEC_ID_MPEG1VIDEO)
      ctx->mb_decision = 2;


    if(formatCtxOut->oformat->flags & AVFMT_GLOBALHEADER)
      ctx->flags |= CODEC_FLAG_GLOBAL_HEADER;
  }

  if(streamOut) {
    AVCodecContext *ctx;
    ctx = streamOut->codec;

    if(avcodec_open2(ctx, codecOut, NULL) < 0)
      return;
  }

  if(!(formatCtxOut->flags & AVFMT_NOFILE))
    if(avio_open(&formatCtxOut->pb, filename, AVIO_FLAG_WRITE) < 0)
      return;

  avformat_write_header(formatCtxOut, NULL);


  // doit

  pktIn = new AVPacket;
  pktOut = new AVPacket;
  av_init_packet(pktOut);
  pktOut->data = 0;

  frame = avcodec_alloc_frame();
  if(!frame)
    return;

  for(;;) {
    if(av_read_frame(formatCtxIn, pktIn) >= 0) {
      av_dup_packet(pktIn);

      int fff;
      if(avcodec_decode_video2(codecCtxIn, frame, &fff, pktIn) < 0)
        std::cout << "bad frame" << std::endl;

      if(!fff)
        return;  // ok

      static int counter=0;
      SaveFrame(frame, codecCtxIn->width, codecCtxIn->height, counter++);  // work fine

      // here a segmentation fault is occured.
      if(avcodec_encode_video2(streamOut->codec, pktOut, frame, &fff) < 0)
        std::cout << "bad frame" << std::endl;
    }
  }
}


// only for testing
// add to ensure frame is valid
void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame) {
  FILE *pFile;
  char szFilename[32];
  int y;

  // Open file
  sprintf(szFilename, "frame%d.ppm", iFrame);
  pFile=fopen(szFilename, "wb");
  if(pFile==NULL)
      return;

  // Write header
  fprintf(pFile, "P6\n%d %d\n255\n", width, height);

  // Write pixel data
  for(y=0; y<height; y++)
      fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);

  // Close file
  fclose(pFile);
}

What am i doing wrong?

While debugging i didn't found any problems with the parameters. streamOut->codec is filled. pktOut is allocated and frame is filled with the picture encoded before. I think the problem is while creating the output codec but watching the example and looking to the doxypages it seems to be correct.

The trace route is from QT using msvc11 and framework 5.

I also tried to run with dr. memory and get this:

Error #26: UNADDRESSABLE ACCESS: reading 0x00000000-0x00000004 4 byte(s)
# 0 replace_memcpy                      [d:\derek\drmemory\withwiki\trunk\drmemory\replace.c:203]
# 1 avcodec-54.dll!ff_dct_quantize_c   +0xd463   (0x6a482364 <avcodec-54.dll+0x3c2364>)
# 2 avcodec-54.dll!avcodec_encode_video2+0xb7     (0x6a56a5b8 <avcodec-54.dll+0x4aa5b8>)
# 3 nmain                               [d:\prg\tests\recording system-qt\libav\recsys\main.cpp:610]
# 4 main                                [d:\prg\tests\recording system-qt\libav\recsys\main.cpp:182]
Note: @0:00:06.318 in thread 5312
Note: instruction: mov    (%edx) -> %ebx

It seems like the reading process while memcpy is going wrong.

---

Version:

I've forgot to mention the version of libav/ffmpeg i'm using:

libavutil      51. 76.100 / 51. 76.100
libavcodec     54. 67.100 / 54. 67.100
libavformat    54. 33.100 / 54. 33.100
libavdevice    54.  3.100 / 54.  3.100
libavfilter     3. 19.103 /  3. 19.103
libswscale      2.  1.101 /  2.  1.101
libswresample   0. 16.100 /  0. 16.100
libpostproc    52.  1.100 / 52.  1.100

---

Addendum:

Function SafeFrame is copied from tutorial 1.

Answer 1

Finally i solved my problem.

The problem is (apart from the documentation of libav) avpacket is not a (real) copy of the picture in the packet. it just points to the data of the packet. You have to make a copy, or better you have to let it libav do.

So first i created a new avframe for the output and a buffer on which the output avframe is pointing to.

AVFrame *outpic = avcodec_alloc_frame();
nbytes = avpicture_get_size(codecCtxOut->pix_fmt, codecCtxOut->width, codecCtxOut->height);
uint8_t* outbuffer = (uint8_t*)av_malloc(nbytes);

This buffer is used for the conversion from input to output. Then in the loop i have to fillup the outpic (avframe) with the buffer. I have found in the code that this function is filling up the plane pointers with the buffer. see here

avpicture_fill((AVPicture*)outpic, outbuffer, AV_PIX_FMT_YUV420P, codecCtxOut->width, codecCtxOut->height);

Then i converted the inpic to outpic using sws_scale. But first you have to setup the swscontext.

SwsContext* swsCtx_ = sws_getContext(codecCtxIn->width, codecCtxIn->height,
                                     codecCtxIn->pix_fmt,
                                     codecCtxOut->width, codecCtxOut->height,
                                     codecCtxOut->pix_fmt,
                                     SWS_BICUBIC, NULL, NULL, NULL);

sws_scale(swsCtx_, inpic->data, inpic->linesize, 0, codecCtxIn->height, outpic->data, outpic->linesize);

Then you can encode the outpic into pktout (avpacket for output). But first do free the output packet, otherwise you will get an error and a leak... see here

av_free_packet(pktOut);

if(avcodec_encode_video2(streamOut->codec, pktOut, outpic, &fff) < 0) {
  std::cout << "shit frame" << std::endl;
  continue;
}
// and write it to the file
formatOut->write_packet(formatCtxOut, pktOut);

So now it works (nearly fine) for me. Still a small memory leak, but this i can spot later.