Decoding a h264 (High) stream with OpenCV's ffmpeg on Ubuntu

Question

I am working with a video stream from an ip camera on Ubuntu 14.04. Everything was going great with a camera that has these parameters (from FFMPEG):

    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 352x192, 29.97 tbr, 90k tbn, 180k tbc

But then i changed to a newer camera, which has these parameters:

    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1280x720, 25 fps, 25 tbr, 90k tbn, 50 tbc

My C++ program uses OpenCV3 to process the stream. By default OpenCV uses ffmpeg to decode and display the stream with function VideoCapture.

VideoCapture vc;
vc.open(input_stream);
while ((vc >> frame), !frame.empty()) {
   *do work*
}

With the new camera stream i get errors like these (from ffmpeg):

[h264 @ 0x7c6980] cabac decode of qscale diff failed at 41 38
[h264 @ 0x7c6980] error while decoding MB 41 38, bytestream (3572)
[h264 @ 0x7c6980] left block unavailable for requested intra mode at 0 44
[h264 @ 0x7bc2c0] SEI type 25 truncated at 208

The image sometimes is glitched, sometimes completely frozen. However on vlc it plays perfectly. I installed the newest version (3.2.2) of ffmpeg player with

./configure --enable-gpl --enable-libx264

Now playing directly with ffplay (instead of launching from source code with OpenCV function VideoCapture), the stream plays better, but sometimes still displays warnings:

[NULL @ 0x7f834c008c00] SEI type 25 size 896 truncated at 320=1/1   
[h264 @ 0x7f834c0d5d20] SEI type 25 size 896 truncated at 319=1/1   
[rtsp @ 0x7f834c0008c0] max delay reached. need to consume packet   
[rtsp @ 0x7f834c0008c0] RTP: missed 1 packets
[h264 @ 0x7f834c094740] concealing 675 DC, 675 AC, 675 MV errors in P frame

Changing the camera hardware is not an option. The camera can be set to encode to h265 or mjpeg. When encoding to mjpeg it can output 5 fps, which is not enough. Decoding to a static video is not an option either, because i need to display real time results about the stream. Here is a list of API backends that can be used in function VideoCapture. Maybe i should switch to some other decoder and player? From my research i conclude that i have these options:

• Somehow get OpenCV to use libVlc instead of ffmpeg

One example of switching to vlc is here, but i don't understand it well enough to say if that is what i need. Or maybe i should be parsing the stream in code?

• Use vlc to preprocess the stream, as suggested here.

This is probably slow, which again is bad for real time results.
Any suggestions and coments will be appreciated.

Answer 1

The errors are caused by packet loss since it uses RTP by default. You are seeing more errors now because you have switched to a higher bitrate input.

Append ?tcp to your input, eg. rtsp://*private*/media/video2?tcp in OpenCV's open function to force TCP, assuming it's supported by your hardware and/or usage scenario.

Answer 2

I have mostly solved this issue.

• First i recompiled OpenCV with the newest version of ffmpeg manually installed on my device with the necessary settings.
• I checked the stream with VLC tools > codec information to check if the stream has corrupted frames, which it didn't.
• I lowered the stream resolution. This gave the biggest improvement.

• The last error that i'm still left with is

  [NULL @ 0x7f834c008c00] SEI type 25 size 896 truncated at 320=1/1
  [h264 @ 0x7f834c0d5d20] SEI type 25 size 896 truncated at 319=1/1

but it doesn't visibly damage the frames. I have not resolved the stream freezing issue however, but it has something to do with my code, not the software. If i can help anyone with similar problems, feel free to ask additional info.