*

ipcpipeline: Splitting a GStreamer pipeline into multiple processes

Posted on 17/11/2017 by George Kiagiadakis

Earlier this year I worked on a certain GStreamer plugin that is called “ipcpipeline”. This plugin provides elements that make it possible to interconnect GStreamer pipelines that run in different processes.  In this blog post I am going to explain how this plugin works and the reason why you might want to use it in your application.

Why ipcpipeline?

In GStreamer, pipelines are meant to be built and run inside a single process. Normally one wouldn’t even think about involving multiple processes for a single pipeline. You can (and should) involve multiple threads, of course, which is easily done using the queue element, in order to do parallel processing. But since you can involve multiple threads, why would you want to involve multiple processes as well?

Splitting part of a pipeline to a different process is useful when there is one or more elements that need to be isolated for security reasons. Imagine the case where you have an application that uses a hardware video decoder and therefore has device access privileges. Also imagine that in the same pipeline you have elements that download and parse video content directly from a network server, like most Video On Demand applications would do. Although I don’t mean to say that GStreamer is not secure, it can be a good idea to think ahead and make it as hard as possible for an attacker to take advantage of potential security flaws. In theory, maybe someone could exploit a bug in the container parser by sending it crafted data from a fake server and then take control of other things by exploiting those device access privileges, or cause a system crash. ipcpipeline could help to prevent that.

How does it work?

In the – oversimplified – diagram below we can see how the media pipeline in a video player would look like with GStreamer:

image.YNGV9Y.png

With ipcpipeline, this pipeline can be split into two processes, like this:

image.WCEG9Y.png

As you can see, the split mainly involves 2 elements: ipcpipelinesink, which serves as the sink for the first pipeline, and ipcpipelinesrc, which serves as the source for the second pipeline. These two elements internally talk to each other through a unix pipe or socket, transferring buffers, events, queries and messages over this socket, thus linking the two pipelines together.

This mechanism doesn’t look very special, though. You might be wondering at this point, what is the difference between using ipcpipeline and some other existing mechanism like a pair of fdsink/fdsrc or udpsink/udpsrc or RTP? What is special about these elements is that the two pipelines behave as if they were a single pipeline, with the elements of the second one being part of a GstBin in the first one:

image.9EBV9Y.png

The diagram above illustrates how you can think of a pipeline that uses the ipcpipeline mechanism. As you can see, ipcpipelinesink behaves as a GstBin that contains the whole remote pipeline. This practically means that whenever you change the state of ipcpipelinesink, the remote pipeline’s state changes as well. It also means that all messages, events and queries that make sense are forwarded from one pipeline to the other, trying to implement as closely as possible the behavior that a GstBin would have.

This design practically allows you to modify an existing application to use this split-pipeline mechanism without having to change the pipeline control logic or implement your own IPC for controlling the second pipeline. It is all integrated in the mechanism already.

ipcpipeline follows a master-slave design. The pipeline that controls the state changes of the other pipeline is called the “master”, while the other one is called the “slave”. In the above example, the pipeline that contains the ipcpipelinesink element is the “master”, while the other one is the “slave”. At the moment of writing, the opposite setup is not implemented, so it’s always the downstream part of the pipeline that can be slaved and ipcpipelinesink is always the “master”.

While it is possible to have only one “master” pipeline, it is possible to have multiple “slave” ones. This allows, for example, to split an audio decoder and a video decoder into different processes:

ipcpipeline-1.png

It is also possible to have multiple ipcpipelinesink elements connect to the same slave pipeline. In this case, the slave pipeline will follow the state that is closest to PLAYING between the two states that it will get from the two ipcpipelinesinks. Also, messages from the slave pipeline will only be forwarded through one of the two ipcpipelinesinks, so you will not notice any duplicate messages. Behavior should be exactly the same as in the split slaves scenario.

ipcpipeline-2.png

Where is the code?

ipcpipeline is part of the GStreamer bad plugins set (here). Documentation is included with the code and there are also some examples that you can try out to get familiar with it. Happy hacking!

 

Original post

Comments (2)

  1. jmz:
    Mar 14, 2018 at 06:56 AM

    Thank you for introducing and explaining the ipcpipeline. I am interested in the last scenario. In Process 1, how do you create a slave pipeline consisting of actually two pipelines (one is audio sink and the other is video sink)? Do the audio pipeline and video pipeline run in separate main loops? Does Process 1 consist of two slave pipelines actually?

    Reply to this comment

    Reply to this comment

    1. gkiagia:
      Mar 14, 2018 at 01:48 PM

      Hi,
      In process 1, as it is shown in the image, there is only one slave pipeline, with one main loop. What happens in that pipeline is that we add two sources and link them to their respective downstream elements all the way to the sinks. All the elements are still in one slave pipeline, but in two separate chains that don't link to each other.

      Reply to this comment

      Reply to this comment


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Latest Blog Posts

GStreamer spring in Sweden

18/05/2018

Earlier this month, Olivier Crête, Nicolas Dufresne, George Kiagiadakis & I attended the GStreamer Spring Hackfest in Lund, Sweden. Hosted…

GPU virtualization update

09/05/2018

A few months ago, Robert Foss wrote a blog post about virtualizing GPU Access. Here's a look at some of the major improvements that have…

GStreamer Spring Hackfest

01/05/2018

Generously hosted by Axis in the beautiful Lund, Sweden, the annual spring hackfest is an occasion for the community to get together to…

Upstream Linux support for new NXP i.MX 8

17/04/2018

The i.MX 6 platform has for the past few years enjoyed a large effort to add upstream support to Linux and surrounding projects. The newly…

A new era for Linux's low-level graphics - Part 2

23/03/2018

Following on from part 1 in the series, part 2 covers more developments in low-level graphics, including support for buffer modifiers in…

A new era for Linux's low-level graphics - Part 1

20/03/2018

The latest enhancements to the DRM subsystem have made mainline Linux much more attractive, making drivers easier to write, applications…

Open Since 2005

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2018. All rights reserved. Website sitemap.