On the fly transcoding and HLS streaming with ffmpeg

Question

I am building a web application that involves serving various kinds of video content. Web-friendly audio and video codecs are handled without any problems, but I am having trouble designing the delivery of video files incompatible with HTML5 video players like mkv containers or H265.

What I have done till now, is use ffmpeg to transcode the video file on the server and make HLS master and VOD playlists and use hls.js on the frontend. The problem, however, is that ffmpeg treats the playlist as a live stream playlist until transcoding is complete on the whole file and then it changes the playlist to serve as VOD. So, the user can't seek until the transcoding is over, and that my server has unnecessarily transcoded the whole file if the user decides to seek the video file halfway ahead. I am using the following ffmpeg command line arguments

ffmpeg -i sample.mkv \
       -c:v libx264 \
       -crf 18 \
       -preset ultrafast \
       -maxrate 4000k \
       -bufsize 8000k \
       -vf "scale=1280:-1,format=yuv420p" \
       -c:a copy -start_number 0 \
       -hls_time 10 \
       -hls_list_size 0 \
       -f hls \
file.m3u8

Now to improve upon this system, I tried to generate the VOD playlist through my app and not ffmpeg, since the format is self explanatory. The webapp would generate the HLS master and VOD playlists beforehand using the video properties such as duration, resolution and bitrate (which are known to the server) and serve the master playlist to the client. The client then starts requesting the individual video segments at which point the server will individually transcode and generate each segment and serve them. Seeking would be possible as the client already has the complete VOD playlist and it can request the specific segment that the user seeks to. The benefit, as I see it, would be that my server would not have to transcode the whole file, if the user decides to seek forward and play the video halfway through.

Now I tried manually creating segments (10s each) from my sample.mkv using the following command

ffmpeg -ss 90 \
       -t 10 \
       -i sample.mkv \
       -g 52 \
       -strict experimental \
       -movflags +frag_keyframe+separate_moof+omit_tfhd_offset+empty_moov \
       -c:v libx264 \
       -crf 18 \
       -preset ultrafast \
       -maxrate 4000k \
       -bufsize 8000k \
       -vf "scale=1280:-1,format=yuv420p" \
       -c:a copy \
fileSequence0.mp4

and so on for other segments, and the VOD playlist as

#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:4
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10.0,
fileSequence0.mp4
#EXTINF:10.0,
fileSequence1.mp4
...
... and so on 
...
#EXT-X-ENDLIST

which plays the first segment just fine but not the subsequent ones.

Now my questions,

1. Why don't the subsequent segments play? What am I doing wrong?

2. Is my technique even viable? Would there be any problem with presetting the segment durations since segmenting is only possible after keyframes and whether ffmpeg can get around this?

My knowledge regarding video processing and generation borders on modest at best. I would greatly appreciate some pointers.

Answer 1

It is possible, but it is very difficult. I would even argue that it may not be possible with ffmpeg. Transport streams have time stamps and continuity counters, these values should be preserved across segment boundaries. The -copyts flag may help a little with that. B frames are extreamly difficult to handle in this case because they will end up with timestamps outside the segment. Audio is difficult as well. Audio has priming samples when the encoder is initialized meaning you may have extra samples every segment That come through as audio pops.

TLDR, is possible, but you need to understand of the container and underlying codecs are structured, and work with them.