gstreamer hangs when encoding a splitted stream with h264

Question

I am using gstreamer to build a pipeline which splits an incoming video stream into two branches, one of which is displayed on the screen, and the other one is encoded and saved on the disk. However, the pipeline hangs after a few frames, and I fail to understand why. To reproduce the issue, just run the following

gst-launch-1.0 videotestsrc ! tee name=t t. ! queue ! autovideosink t.! queue ! x264enc ! matroskamux ! filesink location=sample.mkv

I found something that might be relevant here https://gstreamer.freedesktop.org/documentation/x264/index.html?gi-language=c

In particular

Some settings, including the default settings, may lead to quite some latency (i.e. frame buffering) in the encoder. This may cause problems with pipeline stalling in non-trivial pipelines, because the encoder latency is often considerably higher than the default size of a simple queue element. Such problems are caused by one of the queues in the other non-x264enc streams/branches filling up and blocking upstream. They can be fixed by relaxing the default time/size/buffer limits on the queue elements in the non-x264 branches, or using a (single) multiqueue element for all branches. Also see the last example below. You can also work around this problem by setting the tune=zerolatency property, but this will affect overall encoding quality so may not be appropriate for your use case.

However, I still do not understand how a queue put on a different branch can affect what x264enc is doing.

Answer 1

Understand what PREROLLING in GStreamer means. (TL;DR, the pipeline will wait for each sink to have at least one buffer ready before it can switch to PLAYING)

Also keep in mind that data buffers are passed around via reference, not copies. Now think about what happens with lookahead in the encoder (TL;DR, it will consume frames without outputting enything, until enough lookahead frames have entered the encoder. The diplay brnach howver will not consume anything)

If the display sink does not consume anything it will block upstream, because where would the data go elsewhere. If upstream is blocked, it cannot deliver anymore data to the encoder which it requires to unblock itself. Hence everything stalls.