Capture images from a gstreamer Pipeline

Question

I have a IP camera that uses the RTSP protocol to transmit images, the following code uses gstreamer to connect, pick up those images and show in Swing (works just right).

What I want to do is pick up the frames direct from the Pipeline of gstreamer (not use Swing), so that I can analyze frame by frame the image.

import java.awt.BorderLayout;
import java.awt.Dimension;
import javax.swing.JFrame;
import javax.swing.SwingUtilities;
import org.gstreamer.Element;
import org.gstreamer.Gst;
import org.gstreamer.Pipeline;
import org.gstreamer.State;
import org.gstreamer.swing.VideoComponent;

public class Main {

    public static void main(String[] args) throws InterruptedException {
        args = Gst.init("PipelineLauncher", args);
        final String def = "rtspsrc location=rtsp://192.168.25.160/av0_0 latency=0 ! decodebin ! ffmpegcolorspace name=testp";
        final Pipeline pipe = Pipeline.launch(def);

        SwingUtilities.invokeLater(new Runnable() {

            @Override
            public void run() {
                // Create the video component and link it in                
                VideoComponent videoComponent = new VideoComponent();
                Element videosink = videoComponent.getElement();
                pipe.add(videosink);
                pipe.getElementByName("testp").link(videosink);
                pipe.setState(State.PAUSED);

                if (pipe.isPlaying()) {
                    System.out.println("Pipeline playing");
                } else {
                    System.out.println("Pipeline not playing");
                }

                // Start the pipeline processing
                pipe.play();
                pipe.setState(State.PLAYING);

                if (pipe.isPlaying()) {
                    System.out.println("Pipeline playing");
                } else {
                    System.out.println("Pipeline not playing");
                }

                // Now create a JFrame to display the video output
                JFrame frame = new JFrame("Swing Video Test");
                frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
                frame.add(videoComponent, BorderLayout.CENTER);
                videoComponent.setPreferredSize(new Dimension(800, 600));
                frame.pack();
                frame.setLocationRelativeTo(null);
                frame.setVisible(true);
            }
        });

        Gst.main();
        pipe.setState(State.NULL);
    }

}

Answer 1

I suggest using a pad probe (http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-add-probe). It will then trigger a callback with the buffer that is about to go through the pad. So you can put this at the pad you are most interested in in the pipeline (encoded RTP data or uncompressed raw image for example).

If you are just interested in the final data the appsink is also a good place to look at (http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-appsink.html).

In case of appsink and if you want to display the image and also want the data you may use a tee element (http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-tee.html).