Gstreamer - How to structure application with multiple src's?

Question

I have a situation where I have multiple cameras (rtspsrc), and a singleton element, that does analytics on the incoming video stream. I call it a singleton element, because it has request source and sink pads. Only one of them should exist in the application, because it does it's work on the GPU, and can get better performance by doing things in batch. Think of the application I'm building as an API to add cameras, remove cameras, turn analytics on and off per camera, etc. Cameras will have analytics done on them, capturing the results, and sending them onwards. The complication being, I need to share a Gstreamer element (the analytics element).

So I have multiple cameras, feeding into this single element, then feeding out, into appsinks. This works reasonably well, but I want to be able to:

• Pause a specific camera
• Have each rtspsrc be completely isolated, so errors in one, don't affect the entire pipeline
• Listen for events on a particular camera

If I have all the cameras in a pipeline together, I cannot figure out how to pause a specific camera. I cannot pause the entire pipeline, because that will stop all cameras. The best I've come up with is to remove and unlike the elements for a specific cameras, then when resuming, re-add and re-link. This works sort of. If a specific rtspsrc stops responding, then the entire pipeline stops. If a specific rtspsrc doesn't exist then the entire pipeline won't transition to PLAYING state

How should I architect my application? Do you think I should have a single big pipeline? Or should I have a pipeline containing the singleton analytics element, and a pipeline per camera, then connect them using appsink and appsrc? This approach might make it easier to handle things, as each pipeline is entirely separate?

Let me know if you need more info.

Answer 1

Monolithic architectures should generally be avoided in programming, and your scenario is no exception to this. You have already experienced some complications of managing everything in one pipeline, and the found workarounds will likely cause more problems down the road, plus they do not provide convenient access for managing each camera.

I would therefore recommend to take the second approach to have a pipeline per camera, and additionally implement queues for buffering with an architecture similar to the one from this SO answer. You may also want to ensure your singleton is thread safe to avoid any race conditions between the pipelines when analytics are sent from the cameras.

Answer 2

Given your requirements, I would build most of the API, camera management and GUI in c#, with an MVVM pattern and a DI container, so you decouple as much as possible your API parts, and make them as testable as possible. One other motivation is that it's very quick to produce a UI with this ecosystem (C#, Visual Studio); also for most projects, you know that maintenance will be the main cost Development cost versus maintenance cost, hence decoupling and testing against interfaces is excellent to keep those costs as low as possible.; MVVM will allow you to test your UI Writing a Testable Presentation Layer with MVVM. Decoupling your software components will also allow you to upgrade a certain implementation without touching the rest, and compose your software with its components in the composition root. Often such practices allow you to start with tests (TDD).

I would make sure to have 1 pipeline per camera, to simplify resource management, and if you use cudastreams (cudastreams to simplify concurrency), you can have multiple tasks of video analytics on one GPU, each stream doing the analytics of one camera video stream. You might want to use some proven code from opencv and make sure that it is transposable in a cudastream. If the amount of data, your performance requirements and your hardware won't need/allow such thing, you can just use one opencv processing.

On the native part, (gstreamer), it's relatively easy to interface your components with c# with interop; for instance:

extern "C" __declspec(dllexport) auto Query(myClass* p, const wchar_t* somePath)
    -> structResult*
{ return p->Query(somePath); }

and on the managed part:

[DllImport("myAssembly.dll", CallingConvention = CallingConvention.Cdecl)]
internal static extern IntPtr Query(IntPtr myClassPointer, IntPtr somePath);