What effect does the media type string inserted in a gstreamer pipeline have

Question

I have seen this kind of pipeline-running commands in gstreamer:

e.g.,

gst-launch-1.0 videotestsrc ! video/x-raw, format=I420, framerate=25/1, width=640, height=360 ! xvimagesink

And I have read in some pages that video/x-raw, format=I420, framerate=25/1, width=640, height=360 specifies the media-type. But I am not able to understand what effect would it make - is it transforming the input to the specified framerate/format/width/height etc.. or is it just like specifying the input is in this framerate/width/ht already? And what effect it would have on the pipeline if it is just specifying the input is in this framerate etc... instead of transforming.

And is it really necessary or we can ignore it?

Answer 1

They call this video/x-raw, format=I420, framerate=25/1, width=640, height=360 thing "capabilities" or "caps" for short.

Sometimes it is required to specify explicitly what type of data flows between elements in a gstreamer pipe. And caps is the way to do this.

Why this is required sometimes? Because some gstreamer elements can accept (or produce) several types of media. You can see this with gst-inspect command:

$ gst-inspect videotestsrc
...

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw-yuv
                 format: YUY2
           color-matrix: { sdtv, hdtv }
            chroma-site: { mpeg2, jpeg }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw-yuv
                 format: UYVY
           color-matrix: { sdtv, hdtv }
            chroma-site: { mpeg2, jpeg }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw-yuv
                 format: YVYU
           color-matrix: { sdtv, hdtv }
...

This means that videotestsrc has pad src which can produce outputs in various formats (YUY2, UYVY, YVYU, etc), sizes, framerates, etc.

The same is for xvimagesink:

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw-rgb
              framerate: [ 0/1, 2147483647/1 ]
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
      video/x-raw-yuv
              framerate: [ 0/1, 2147483647/1 ]
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]

It is able to view streams of data in various formats.

GStreamer uses the process called caps negotiation to decide which concrete data format to use between two elements. It tries to select "best fitting" format if no capabilities are provided by the user. That is why it is possible to drop capabilities from your pipeline and it will still work:

$ gst-launch-1.0 -v videotestsrc ! xvimagesink

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)320, height=(int)240, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)320, height=(int)240, framerate=(fraction)30/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

I added -v flag to see what caps gstreamser actually decided to use.

But sometimes there are cases when gstreamser fails to negotiate data format or you have different preferences.

E.g. in the case of pipeline that reads stream from socket it is impossible guess data format for certain and you need to provide correct capabilities.

You can see that specifying caps makes difference by executing these two pipelines:

$ gst-launch -v videotestsrc ! 'video/x-raw-yuv, width=600, height=600' ! xvimagesink
$ gst-launch -v videotestsrc ! 'video/x-raw-yuv, width=60, height=60' ! xvimagesink

It is important to understand that capabilities does not convert data but specify which format elements will produce or consume.