How to generate "moov before mdat" MP4 video files with high performance (minimal I/O)?

Question

I'm designing a server application that stores a live H.264 video stream as MP4 for later consumption by browsers. Since a server will need to handle as many simultaneous streams as possible, I believe I/O will be the natural bottleneck and I'd like to keep I/O to a minimum. I'm running into the classic MP4 moov/mdat ordering problem: MP4 generators prefer to write the mdat box (containing the actual media frames) first, then write the moov box (containing file offsets and other structural information) later, after it actually knows what the mdat file offsets are. MP4 consumers prefer the opposite for progressive streaming -- reading the moov box first, so the mdat structure is known and the video can start playing quickly without needing to download the whole file.

The usual solution is to post-process MP4 files to move the moov box ahead of the mdat box, and rewriting file offsets accordingly. However, for a high-volume application, I'd like to avoid the I/O penalty of writing the incoming video data to disk, reading it all back in, and writing it again with a new arrangement.

Several approaches come to mind:

1. Post-process the MP4 as usual, incurring the I/O penalty and potentially delaying the video's availability. (not good.)
2. Use fragmented MP4 and small fragment sizes that are reasonable to fit in memory. (This could have negative effects on whole-file seekability, I'd think.)
3. It would be awesome if filesystems provided a fast "prepend" option to add new blocks to the beginning of a file's block chain. (I don't think this has been invented yet.)
4. Generate the MP4 as two files -- an "mdat" file (containing the actual media frames) and a "moov" file (containing the ftyp header and moov data). These two files, if concatenated, would produce a valid moov-first MP4 file. A simple web server module could present a virtual .mp4 file to the user, but read the .moov and .mdat files behind the scenes.

I'm leaning towards #4 for now. Is there any more practical way to solve this problem?

Answer 1

If the size of the moov data is estimable, preallocate space at the beginning of the file. Some of it might be wasted, but you won't have to recalculate any offsets, and it will avoid the I/O costs in some percentage of cases. Just make sure you have a fallback when the moov data gets bigger than your estimate.