Haskell GStreamer tee element(1-N) trouble

Question

The problem i have relates to the following piece of code:

module Main(main) where

import qualified Media.Streaming.GStreamer as GS
import Data.Maybe
import System.IO
import System.Exit
import System.Glib.MainLoop as Glib
import System.Glib.Signals as Glib
import System.Glib.Properties as Glib


makeElement:: String → String → IO GS.Element
makeElement elementType elementName = do
    element ← GS.elementFactoryMake elementType (Just elementName)
    case element of
        Just element' → return element'
        Nothing → do
            hPutStrLn stdout ("Cannot create element!")
            hFlush stdout
            exitFailure

player =  do
    GS.init

    pipeline ← GS.pipelineNew "video-stream"

    source  ← makeElement "v4l2src" "video-source"
    color   ← makeElement "ffmpegcolorspace" "video-color"
    tee     ← makeElement "tee" "stream-tee"
    rQ      ← makeElement "queue" "record-queue"
    vQ      ← makeElement "queue" "video-queue"
    encoder ← makeElement "y4menc" "video-encoder"
    rSink   ← makeElement "filesink" "record-sink"
    sink    ← makeElement "ximagesink" "video-sink"

    let elements = [source,color,encoder,rSink,vQ,rQ,sink,tee]

    Glib.objectSetPropertyString "location" rSink "rec"

    mapM_ (GS.binAdd (GS.castToBin pipeline)) elements

    -- Request Pads from tee
    dPad ← GS.elementGetRequestPad tee "src%d"
    rPad ← GS.elementGetRequestPad tee "src%d"
    -- Request Static Pads from queue
    sDPad ← GS.elementGetStaticPad vQ "sink"
    sRPad ← GS.elementGetStaticPad rQ "sink"
    -- Link tee source to queue sink
    GS.padLink (fromJust dPad) (fromJust sDPad)
    GS.padLink (fromJust rPad) (fromJust sRPad)

    GS.elementReleaseRequestPad tee $ fromJust dPad
    GS.elementReleaseRequestPad tee $ fromJust rPad

    GS.elementLink source color
    GS.elementLink color tee
    GS.elementLink vQ sink
    GS.elementLink rQ encoder
    GS.elementLink encoder rSink


    GS.elementSetState pipeline GS.StatePlaying

main = do
    loop ← Glib.mainLoopNew Nothing False
    player
    Glib.mainLoopRun loop

The code compiles fine, camera LED switches ON and the file is created but then NOTHING. Without the tee and queue elements, the separate setup for recording/displaying video works just fine.Also, the same pipeline works perfectly if i test it with gst-launch. I'm missing something here on how gstreamer works but i can't figure out what.

Also, if it helps, i'm building on ArchLinux using:
- GHC 7.0.3 ;
- gstreamer-bindings 0.12.1 ;
- gtk2hs 0.12.2 ;
- gstreamer 0.10.35-1 ;
- glib 1.2.10-9 .

Answer 1

RESOLVED

I found my solution, and what follows is a lengthy post but please, bear with me. I must share my frustration with someone.

After many more buggy tries i decided to go back to testing some setups using gst-launch. This helped me to find out that after the queue element that buffers the part which goes to the filesink i needed another ffmpegcolorspace element to setup the correct video format i think. At this point i was not going back to trying this thing out it Haskell again, i thought i needed to get 'closer' so i decided to try it in C. As a side note , i don't know C , i can understand the syntax but that's about it...and for goodness sake i'm just now trying to learn Haskell. To continue, i decided to also try using 'GS.elementGetCompatiblePad' on the tee element so i can be sure that the pads will link with the queue.

The C code i stitched together is this :

#include <gst/gst.h>
#include <glib.h>

int
main (int argc,char *argv[])
{

    GstElement *pipeline, *source, *color, *color2 , *color3, *tee, *rQ, *vQ, *encoder,   *fSink , *sink;
    GMainLoop *loop;
    loop = g_main_loop_new (NULL,FALSE);
    /* initialize gstreamer */
    gst_init(&argc,&argv);

    /* creating elements */
    pipeline = gst_pipeline_new("stream-pipeline");

    source = gst_element_factory_make ("v4l2src","stream-source");
    color = gst_element_factory_make ("ffmpegcolorspace","video-color");
    tee = gst_element_factory_make ("tee","stream-tee");
    rQ = gst_element_factory_make ("queue","record-queue");
    vQ = gst_element_factory_make ("queue","video-queue");
    encoder = gst_element_factory_make ("theoraenc","video-encoder");
    fSink = gst_element_factory_make ("filesink","record-sink");
    sink = gst_element_factory_make ("ximagesink","video-sink");
    color2 = gst_element_factory_make ("ffmpegcolorspace","video-color2");
    color3 = gst_element_factory_make ("ffmpegcolorspace","video-color3");
    /*check that the elements were created */

    if (!source || !color || !tee || !rQ || !vQ || !encoder || !fSink || !sink){
        g_printerr("One element could not be created!");
        return -1;
    }
    /*set file output location */
    g_object_set(G_OBJECT (fSink),"location","rec",NULL);

    gst_bin_add_many (GST_BIN(pipeline),
                        source,color,color2,color3,tee,rQ,vQ,encoder,fSink,sink,NULL);

    /* get request pads */
    GstPad *dPad, *rPad, *sDPad, *sRPad;

    sDPad = gst_element_get_static_pad(vQ,"sink");
    sRPad = gst_element_get_static_pad(rQ,"sink");
    dPad = gst_element_get_compatible_pad(tee,sDPad,GST_CAPS_ANY);
    rPad = gst_element_get_compatible_pad(tee,sRPad,GST_CAPS_ANY);

    /*link pads*/
    gst_pad_link(dPad,sDPad);
    gst_pad_link(rPad,sRPad);

    /*unref pads */
    gst_object_unref(GST_OBJECT(dPad));
    gst_object_unref(GST_OBJECT(rPad));
    gst_object_unref(GST_OBJECT(sDPad));
    gst_object_unref(GST_OBJECT(sRPad));

    /*link elements */
    gst_element_link(source,tee);
    gst_element_link_many(rQ,color2,encoder,fSink,NULL);
    gst_element_link_many(vQ,color3,sink),NULL;

    /*set the pipeline state to playing */
    gst_element_set_state(pipeline,GST_STATE_PLAYING);

    g_main_loop_run (loop);

    gst_element_set_state(pipeline,GST_STATE_NULL);
    gst_object_unref(GST_OBJECT(pipeline));

    return 0;

}

In order to use 'gst_element_get_compatible_pad' i had to first get static pads from the queue elements so i hand to switch those four related lines. I try it out, and Abracadabra ...oh no, wait ... camera starts, the file is created and a window with the 'video' pops, but a black window that remains black!

No problem i say , run the program with gst-debug-level=5 ( =)) ) yea, right , try reading the whole output.I give up for the moment and i thought maybe it has something to do with the elements in my pipeline not working right together so i code another pipeline in C but this time something more simple just with audio files.
I had the same result so i decided tu debug again, this time with runlevel 3 and i started reading the whole thing,line by line.

Somewhere in there i found this:

trying to link stream-tee:src0 and record-queue:sink
trying to link stream-tee:src0 and video-queue:sink

something nasty is happening here

linked stream-tee:src0 and video-queue:sink,successful
trying to link stream-tee:src0 and record-queue:sink
src stream-tee:src0 was already linked with video-queue:sink

And it gives up!
I guess i must go back using gst_element_get_request_pad, but haven't i tried that already? So i switch back to vim and replace all occurrences of 'gst_element_get_compatible_pad with the request counterpart like so:

sDPad = gst_element_get_static_pad(vQ,"sink");
sRPad = gst_element_get_static_pad(rQ,"sink");
dPad = gst_element_get_request_pad(tee,"src%d");
rPad = gst_element_get_request_pad(tee,"src%d");

I gaze upon this code and i say to myself 'you twit', this is where it all started; take a deep breath ; after all this is what the debugger complains about so i compile , i run, and Voila. I found my solution.

Those four lines had to be reversed, i had to first get a reference to the static pads and then request a reference to a 'request' pad on the tee element.
I go back to haskell a happy man.I implement my solution, compile, fire up,camera starts, the file is created and ... just like that..nothing, not even the black screen.
Filled with anger i just comment out the lines where i release the request pads and decide to compile and run once more, my neck started to hurt a while ago.
Again, by magic it all works , i have video on the screen and in the file.
I guess Haskell just likes to hold tighter and sometimes you have to just go with something that makes no sense. The gstreamer docs state clearly release,release,release.

The final haskell code:

module Main(main) where

import qualified Media.Streaming.GStreamer as GS
import Data.Maybe
import System.Exit
import System.Glib.MainLoop as Glib
import System.Glib.Signals as Glib
import System.Glib.Properties as Glib

makeElement:: String → String → IO GS.Element
makeElement elementType elementName = do
        element ← GS.elementFactoryMake elementType (Just elementName)
        case element of
            Just element' → return element'
            Nothing → do
                    putStrLn "Cannot create element!"
                    exitFailure

linkSPadToStaticSink::(GS.ElementClass object, GS.ElementClass elementT) ⇒ object →     elementT → IO (Glib.ConnectId object)
linkSPadToStaticSink elSrc elSink = do
            Glib.on elSrc GS.elementPadAdded (λpad → do
                                                    sinkPad ← GS.elementGetStaticPad elSink "sink"
                                                    GS.padLink pad (fromJust sinkPad)
                                                    return ∅)

player =  do
        GS.init
        pipeline ← GS.pipelineNew "video-stream"
        source ← makeElement "v4l2src" "video-source"
        color ← makeElement "ffmpegcolorspace" "video-color"
        color2 ← makeElement "ffmpegcolorspace" "video-color2"
        tee ← makeElement "tee" "stream-tee"
        rQ ← makeElement "queue" "record-queue"
        vQ ← makeElement "queue" "video-queue"
        encoder ← makeElement "y4menc" "video-encoder"
        rSink ← makeElement "filesink" "record-sink"
        sink ← makeElement "ximagesink" "video-sink"

        let elements = [source,color,color2,encoder,rSink,vQ,rQ,sink,tee]

        Glib.objectSetPropertyString "location" rSink "rec"

        mapM_ (GS.binAdd (GS.castToBin pipeline)) elements

        -- Get static pads from queue elements
        sDPad ← GS.elementGetStaticPad vQ "sink"
        sRPad ← GS.elementGetStaticPad rQ "sink"
        -- Request pads from tee element
        dPad ← GS.elementGetRequestPad tee "src%d"
        rPad ← GS.elementGetRequestPad tee "src%d"
        -- Link tee source to queue sink
        GS.padLink (fromJust dPad) (fromJust sDPad) 
        GS.padLink (fromJust rPad) (fromJust sRPad)

        GS.elementLink source color
        GS.elementLink color tee
        GS.elementLink vQ sink
        GS.elementLink rQ color2
        GS.elementLink color2 encoder
        GS.elementLink encoder rSink

        GS.elementSetState pipeline GS.StatePlaying

main = do
    loop ← Glib.mainLoopNew Nothing False
    player
    Glib.mainLoopRun loop

Now i ask you, should/could i have seen this ?
Was it that obvious ?

I'm glad this will make me be more careful and look in not so obvious places but...eww.

In conclusion to all this, i learned about the gstreamer debug options, i learned that it whispers to me and i MUST listen. I learned about GDB being forced to used because when i began stitching C code all i got was a 'seg fault'.
I learned to love lazy-eval and pure Haskell code.
A little bit of Haskell, maybe a tiny bit of C and more experience. 'Lost' about half a day, three classes and several hours of sleep but after all...So it goes...