Decoding a proprietary H.264 network video stream?

Question

I'm trying to view this IP camera I got off eBay (yeah, yeah I know). Model BL-5720IPW-L4MM.

Anyway, they come with really crappy Windows based application that's mostly in Chinese, and the functionality is very poor. I've already asked the supplier if I can have the format of the video stream so I may supply them with a Linux alternative - but they declined... Anyway, they were quite expensive and I need to read the Video stream from my Linux server (No X) and plan to integrate the stream in to the linux application 'motion'. But I cannot figure out where the raw video begins, and it what exact format it's streaming.

From what I can gather (in byte offset/length - desc - example):
0/4 Firmware Version - 8.4.4.5
5/2 H-res - 1280
7/2 V-res - 720

And what I know about the stream:
* 264/MPEG4 based codec
* 25fps

Below is the raw data captured from TCP 9001 (It starts streaming as soon as you connect). If ANYONE is able to assist me get a hold on this format, it'd be much appreciated. The full 24MB (~60 second) capture is available here and should show my very messy desk if decoded correctly.

hexdump -n 1024 -C camera-capture.raw
00000000  08 04 04 05 00 05 d0 02  b0 01 f0 00 00 00 00 00  |................|
00000010  00 00 00 00 42 66 00 00  00 f7 0e 0f ff ff ff ff  |....Bf..........|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 01 b6 3f f0  18 54 61 a5 b6 4b c6 db  |......?..Ta..K..|
00000040  7f 1b 6d f8 8c 6d bf 8d  b6 fe 36 db f8 db 6f e3  |..m..m....6...o.|
00000050  6d bf 8d b6 fe 36 db f8  db 6f e3 6d bf 8d b6 fe  |m....6...o.m....|
00000060  36 db f8 db 6f e3 6d bf  8d b6 fe 36 db f8 db 6f  |6...o.m....6...o|
00000070  e3 6d bf 8d b6 fe 36 db  f8 db 6f e3 6d bf 8d b6  |.m....6...o.m...|
00000080  fe 36 db f8 db 6f e3 6d  bf 8d b6 fd 1d 89 01 48  |.6...o.m.......H|
00000090  da 81 fd f2 06 fc 2f 73  19 0f 1b 10 0b d5 7c d7  |....../s......|.|
000000a0  e0 bc e1 ec ce b5 a5 1d  04 65 c2 68 86 14 db 77  |.........e.h...w|
000000b0  06 ce 31 b1 d8 97 8a 13  f4 39 92 8c 0e 2a 60 a8  |..1......9...*`.|
000000c0  87 81 e6 7c 38 bc 1a 8c  85 81 66 5e e6 14 70 60  |...|8.....f^..p`|
000000d0  70 f7 ca 9b 93 ee ea f4  43 93 77 ee e4 48 8e ef  |p.......C.w..H..|
000000e0  f1 b7 9f c7 b3 7d 87 32  ea 8b 1b 40 c1 48 76 cc  |.....}[email protected].|
000000f0  5f 88 86 0c 96 e8 f3 49  98 5c 7f d6 96 37 fe e4  |_......I.\...7..|
00000100  3f ea ee fe 56 cc a2 78  bf cf 9f 91 ae a8 cd 33  |?...V..x.......3|
00000110  1d 7b f9 d7 1a b3 ae 26  46 8f f0 f3 2d 99 ca 59  |.{.....&F...-..Y|
00000120  fe 0d 05 d2 f4 ea 1f f7  43 37 1a 61 b4 14 4f 2a  |........C7.a..O*|
00000130  46 3c 7d 3b 02 2c 13 94  21 5d 65 3d 12 ce a2 d3  |F<};.,..!]e=....|
00000140  77 c2 83 ec a6 11 70 66  e3 67 85 c8 d9 20 8e 33  |w.....pf.g... .3|
00000150  9a 84 6a 99 f1 94 9f 30  2c 81 12 57 7f 0c 1c da  |..j....0,..W....|
00000160  16 f6 b1 dc 5b fe 33 14  bd 5a 52 aa a1 0a 08 43  |....[.3..ZR....C|
00000170  92 7a 45 9e cd 2b b7 bc  41 93 b4 66 12 7d ab 75  |.zE..+..A..f.}.u|
00000180  48 a0 cc 46 dd 13 5a 99  25 5f a8 91 0a e7 8a 1a  |H..F..Z.%_......|
00000190  06 14 ae cf 46 5b 30 aa  1f 81 22 c5 9f 5f 19 8d  |....F[0...".._..|
000001a0  3c 48 f8 f3 7b 91 ee dd  e3 cd bb c6 db 7f 1b ef  |<H..{...........|
000001b0  4b 1e 6d de 36 db b2 3c  fe f8 df 6f e3 6d bf 8d  |K.m.6..<...o.m..|
000001c0  b6 fe 3c db bc 6e 5f b2  3d a5 da 9a 6e 24 9b 78  |..<..n_.=...n$.x|
000001d0  66 9c 60 58 6d 72 a5 8b  83 22 5f 0e 1b d7 57 af  |f.`Xmr..."_...W.|
000001e0  71 c2 f1 1d a5 a7 c4 c6  7a a0 27 2e 33 4c 73 c9  |q.......z.'.3Ls.|
000001f0  78 f7 fa f1 e6 dd e3 6d  bb 23 6d b9 c7 9f df 0f  |x......m.#m.....|
00000200  4b 9b 69 4d 3c 7e 66 c4  38 ef 8d b6 fe 36 db f8  |K.iM<~f.8....6..|
00000210  db 6f e3 6d bf 8d b6 fe  36 db f8 db 6f e3 6d bf  |.o.m....6...o.m.|
00000220  8d b6 fe 36 db f8 db 6f  c4 63 6d fc 6d b7 f1 b6  |...6...o.cm.m...|
00000230  df c6 db 7f 1b 6d fc 6d  b7 f1 b6 df c6 db 7f 1b  |.....m.m........|
00000240  6d fc 6d b7 f1 b6 df c6  db 7f 1b 6d fc 6d b7 f1  |m.m........m.m..|
00000250  b6 df c6 db 7f 1b 6d fc  6d b7 f1 b3 2d 91 51 57  |......m.m...-.QW|
00000260  2e 6c 41 2e 54 d9 bf 41  aa 03 10 51 36 5a 3e df  |.lA.T..A...Q6Z>.|
00000270  03 07 1f 17 53 a2 81 e6  16 ee 07 16 e0 d2 36 54  |....S.........6T|
00000280  bf 0b 53 37 09 3a 30 3b  7e 7a 08 dc f6 16 37 b8  |..S7.:0;~z....7.|
00000290  6b 6b 8c 61 15 86 82 78  66 ae ef 4e a7 12 b3 d5  |kk.a...xf..N....|
000002a0  39 3e e8 d0 6d df b7 5e  39 6f 72 36 df f8 df cd  |9>..m..^9or6....|
000002b0  76 e9 ca 7a 9d ee b3 05  bd cf 91 5b a8 87 46 0d  |v..z.......[..F.|
000002c0  53 88 8f d3 9b 3d 8f 31  51 ff 54 ad 1b dd f4 a1  |S....=.1Q.T.....|
000002d0  2f c3 8b 35 78 66 2c 89  73 f6 a6 3c bd 87 d5 ca  |/..5xf,.s..<....|
000002e0  b3 8b c7 9c 55 99 b0 f2  bf 68 45 84 65 ac 3e b7  |....U....hE.e.>.|
000002f0  0f 48 99 c9 b7 11 0c f0  24 47 c9 fd 56 ef 05 2a  |.H......$G..V..*|
00000300  eb 5d 5e 11 9c 6d 89 f2  47 a1 1d 4c 82 cf 5b ea  |.]^..m..G..L..[.|
00000310  b3 45 26 5a 69 3b 5f 0f  1b a8 43 ca 2a 17 aa 2f  |.E&Zi;_...C.*../|
00000320  9f a5 9f 2a f9 30 c2 f0  56 a1 22 37 55 6c 15 ab  |...*.0..V."7Ul..|
00000330  ca 4c 2d a3 c8 16 d4 bc  f8 fa 51 09 b9 ea 16 e8  |.L-.......Q.....|
00000340  c9 62 aa 42 5a 61 e6 7b  1a 7d 63 8d ff 6b 69 ec  |.b.BZa.{.}c..ki.|
00000350  0f 5f 22 9b 6f eb de 0d  df c9 8c 28 9f fc 91 16  |._".o......(....|
00000360  23 3c 9b 35 7e 13 17 cc  10 3d da 46 69 b8 f6 6f  |#<.5~....=.Fi..o|
00000370  8c b0 dc 3a a9 76 71 18  d4 e9 98 bb 96 91 c7 b3  |...:.vq.........|
00000380  c8 d1 ed f4 91 b6 dd b1  b6 dd e3 6d bb c6 db 7f  |...........m....|
00000390  1b 6d fc 6d bf 38 de df  e3 7f 3f 8d e7 ef 1b 79  |.m.m.8....?....y|
000003a0  fc 6d bf 2c 7d cb bc f2  a1 8f 4e 2d 99 8c 24 d2  |.m.,}.....N-..$.|
000003b0  01 7a bf 16 fb 4c 1a 63  79 5c 2a 1b ab 3f 74 b3  |.z...L.cy\*..?t.|
000003c0  b3 96 c2 6a 24 25 b4 58  df 50 3c fc e3 6d b6 e3  |...j$%.X.P<..m..|
000003d0  6d b9 23 6d bb c6 de f7  8f 36 fe 3f 4f cb 24 ac  |m.#m.....6.?O.$.|
000003e0  fa 3d 4e f1 b6 df c6 db  7f 1b 6d fc 6d b7 f1 b6  |.=N.......m.m...|
000003f0  df c6 db 7f 1b 6d fc 6d  b7 f1 b6 df c6 db 7f 1b  |.....m.m........|

Answer 1

The first 16 bytes are the file header. I trust your reading of the firmware version. After it comes two pairs of 16-bit little-endian numbers. Everything in the file is little-endian except the actual MPEG ES. They are width and height, first of the primary high-resolution video stream and then of the secondary low-resolution video stream. The last 4 bytes are 0's. maybe that's a terminator for the list of streams or maybe they would be audio parameters, You have a 1280x720 primary stream and a 432x240 secondary stream.

The rest of the file is a sequence of packets. Each packet has a 16-byte header.

The first 32-bit word in the packet header is a tag that identifies the stream that the packet belongs to, and also whether the packet contains a key frame. (In addition to the 2 video streams there is also a metadata stream, or maybe 2 metadata streams, which I haven't figured out.)

The second word in the packet header is the length of the packet's payload.

The third word in the packet header is possibly a timestamp. It's monotonically increasing in the video streams, and always 0xffffffff in the metadata stream(s).

The fourth word in the packet header is still a complete mystery.

The 16-byte packet header is immediately followed by the packet payload, which is immediately followed by the next packet's header.

Tag values present in the file are:

0: primary video stream key frame
1: mysterious metadata
2: secondary video stream key frame
0x80: primary video stream non-key frame
0x82: secondary video stream non-key frame
0xc6821001: mysterious metadata

In the video streams, each packet contains a single MPEG4 VOP (frame). Some of them have extra 0 bytes at the beginning, but those are harmless. The VOP starts with the 4-byte sequence 0x00 0x00 0x01 0xb6.

When all the 0 and 0x80 packet payloads are concatenated, or the 2's and 0x82's, the result is almost a valid MPEG4 video stream. There is something slightly weird going on with the key frames. The vop_time_increment field is supposed to be preceded by a 1 bit (the "marker"), but the key frames all have a 0 there making them invalid. That's the minor problem.

The major problem is that the stream has no VOL header. That's where MPEG4 stores basic information about the video like width, height, the tick rate, the interlace flag... all this stuff is missing except the width and height, and those are only contained in the proprietary file header.

But when you're desperate enough, you can just guess at all the flags and make up a fake VOL header. I took the VOL header from a completely unrelated MPEG4 video, altered the width and height to 1280x720, and appended the 0 and 0x80 packets from your camera. The video played!

There are some imperfections. The beginning shows blockiness which clears up eventually. Normally I'd say that's the result of a stream not beginning with a key frame but this one does, so I don't know why it's blocky. And there are flashes of inappropriate brightness once in a while. And the frame rate is way off, not surprising considering the syntax errors near the vop_time_increment fields. The good news is the picture is pretty clear overall.

If the crappy Windows software can export to any standard formats, we might be able to improve our fake header a little bit by studying an exported file header.

Here are the tools I hacked together while decoding this thing. ./streamget < camera.raw | mplayer -demuxer mpeg4es - works for me.

mkvol: write an MPEG4 VOL header to stdout

#!/usr/bin/perl -W
use strict;

@ARGV==2 or die "Usage: $0 width height\n";

my $bits = '';
$bits .= '00000000000000000000000100000000'; # video_object_start_code(0)
$bits .= '00000000000000000000000100100000'; # video_object_layer_start_code(0)
$bits .= '0'; # random_accessible_vol
$bits .= '00010001'; # video_object_type: Advanced Simple Object
$bits .= '1'; # is_object_layer_identifier
$bits .= '0010'; # video_object_layer_verid: 2
$bits .= '000'; # video_object_layer_priority: 0
$bits .= '0001'; # pixel_aspect_ratio: square
$bits .= '0'; # vol_control_parameters
$bits .= '00'; # video_object_layer_shape: rectangular
$bits .= '1'; # marker
$bits .= sprintf '%016b', 30; # ticks_per_sec
$bits .= '1'; # marker
$bits .= '0'; # fixed_vop_rate
$bits .= '1'; # marker
$bits .= sprintf '%013b', $ARGV[0]; # width
$bits .= '1'; # marker
$bits .= sprintf '%013b', $ARGV[1]; # height
$bits .= '1'; # marker
$bits .= '0'; # interlaced
$bits .= '1'; # obmc_disable
$bits .= '00'; # sprite_enable: none # for verid!=1
#$bits .= '0'; # sprite_enable: none # for verid==1
$bits .= '0'; # not_8_bit
$bits .= '0'; # quant_type: second inverse quantization method
$bits .= '0'; # quarter_sample # for verid!=1
$bits .= '1'; # complexity_estimation_disable
$bits .= '1'; # resync_marker_disable
$bits .= '0'; # data_partitioned
$bits .= '0'; # newpred_enable # for verid!=1
$bits .= '0'; # reduced_resolution_vop_enable # for verid!=1
$bits .= '0'; # scalability
$bits .='0'; $bits .= '1' while length($bits)%8; # pad to byte
print pack "B*", $bits;

streamget: extract a video stream from the camera and prepend VOL

#!/usr/bin/perl -W
use strict;

sub fullread;

@ARGV==0 || "@ARGV" eq 'secondary' or die "Usage: $0 [secondary] < camera.raw";

my @wanted = (0,0x80);
@wanted = (2,0x82) if @ARGV;

fullread(my $fileheader, 16, 1);
my ($version, $width, $height, $swidth, $sheight, @mysterystuff) =
  unpack "Nv4C4", $fileheader;
if(!@ARGV) {
  system("./mkvol $width $height");
} else {
  system("./mkvol $swidth $sheight");
}

while(fullread(my $buf, 16)) {
  my ($tag, $plen, $mystery1, $mystery1) = unpack "VVVV", $buf;
  fullread(my $pbuf, $plen, 1);
  if($tag ~~ @wanted) {
    # Uncomment to remove extra 0's between VOPs. They're harmless.
    #$pbuf =~ s/^\x00*(\x00\x00\x01)/$1/;

    # Uncomment this block to fix the bogus marker bits before
    # vop_time_increment. mplayer complains about them but plays the stream
    # anyway. Other players may need this fix.
    #if($pbuf =~ /^\x00\x00\x01\xb6/) {
    #  my $fixit = unpack "B*", substr($pbuf, 4, 32);
    #  $fixit =~ s/(..1*0)0/${1}1/;
    #  substr($pbuf, 4, 32) = pack "B*", $fixit;
    #}
    print $pbuf;
  }
}

# Return false if !$mandatory and EOF was seen immediately.
# Return true if the full amount was read.
# Otherwise die.
sub fullread
{
  my ($buf, $len, $mandatory) = @_;
  my $n = read(STDIN, $_[0], $len);
  if(!defined($n)) {
    die "stdin: $!\n";
  }
  if($n==0) {
    return 0 if !$mandatory;
    die "stdin: unexpected EOF\n"
  }
  if($n<$len) {
    die "stdin: unexpected EOF\n"
  }
  return 1;
}