We're hiring!
*

Run your own CI pipeline with GStreamer's new monorepo

Xavier Claessens avatar

Xavier Claessens
October 26, 2021

Share this post:

Recently, the GStreamer project merged all its git repositories into a single, unified repository, often called monorepo. You can read more about this change here.

One benefit is it greatly simplifies maintaining custom, project specific, GStreamer patches. Previously, projects that needed to develop, or backport, some patches had to go through multiple steps to complete the task. Projects had to fork various git repositories (e.g. gst-plugins-good, gst-plugins-bad, etc), each repository would then have a new branch with the extra commits, and often, gst-build was used to pull all of these repositories together (and gst-build itself had to be patched beforehand to download forked repositories). Thankfully, all that will be a thing of the past.

When maintaining a non-trivial set of patches, it's always a good idea to also run a continuous integration (CI) pipeline. Prior to moving to a single repository, GStreamer's public CI had a complicated manifest mechanism to be able to pull branches from different repositories, and test them together. Replicating that system with a private GitLab instance was close to impossible.

With GStreamer monorepo, all those patches can now be maintained in a single git repository, in a single branch. For example, if you are forking GStreamer's main branch into a private GitLab instance, the CI will now work almost out of the box! Your team can create Merge Requests in your private GitLab instance, and develop features while relying on your own CI. Of course, once the feature is ready, it often makes sense to upstream. A single Merge Request in the public repository and you're good to go.

How does it work?

1. Clone the GStreamer repository:

git clone https://gitlab.freedesktop.org/gstreamer/gstreamer.git

2. Push the main branch to your private GitLab instance:

git remote add private https://gitlab.example.com/my-project/gstreamer.git
git push private main

3. Setup your new GStreamer repository:

  • Navigate to https://gitlab.example.com/my-project/gstreamer/edit
  • Under Visibility, project features, permissions make sure to enable CI/CD and Container registry.

4. Register your GitLab runners:

  • Navigate to https://gitlab.example.com/my-project/gstreamer/-/settings/ci_cd
  • Under Runners you'll find the registration token to register your gitlab-runner.

That's it! You can now run the same CI pipeline as on the public GitLab instance. Every member of your team can create their fork of that repositry into their own account and create internal Merge Requests.

Also, you'll notice that your CI will rebuild the needed Linux and Windows docker images thanks to Freedesktop's ci-template.

Known issues

Not everything is that easy however...

Cerbero trigger job fails

GitLab will try to trigger Cerbero's CI into the public repository, which will fail because your access token from your private GitLab instance won't be accepted by another GitLab instance.

You can either edit .gitlab-ci.yml and delete the cerbero trigger job, or you can also fork cerbero into our private GitLab instance. Note that if you don't fork cerbero under gstreamer/cerbero, you will have to edit ci/gitlab/trigger_cerbero_pipeline.py and change to CERBERO_PROJECT = 'my-project/cerbero'.

I need a Windows runner

You can delete from .gitlab-ci.yml jobs that require Windows, or you can setup your own runner.

  1. Install Windows on a powerful machine Either latest Windows 10 (not a VM), or Windows Server 1809 VM.
  2. Install Docker Desktop https://docs.docker.com/desktop/windows/install/ Make sure to switch it to run Windows Containers instead of Linux Containers.
  3. Install gitlab-runner https://docs.gitlab.com/runner/install/windows.html When registering your runner, make sure to set tags: windows,1809,shell,docker.

Conclusion

This unification of Git repositories will greatly simply the workflow for all contributors and distributors. We are really looking forward to seeing it in action in the next major GStreamer release. With the feature freeze having begun on October 18th, we should hopefully see a final release of GStreamer 1.20 before the end of the year. As usual, if you need any assistance with GStreamer, please don't hesitate to contact us, we're here to help!

Comments (0)


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Visual-inertial tracking for Monado

05/04/2022

Monado now has initial support for 6DoF ("inside-out") tracking for devices with cameras and an IMU! Three free and open source SLAM/VIO…

Spotlight on Meson's full-featured developer environment

30/03/2022

When developing an application or a library, it is very common to want to run it without installing it, or to install it into a custom prefix…

How to write a Vulkan driver in 2022

23/03/2022

An incredible amount has changed in Mesa and in the Vulkan ecosystems since we wrote the first Vulkan driver in Mesa for Intel hardware…

Improving the reliability of file system monitoring tools

14/03/2022

Every file system used in production has tools to try to recover from system crashes. To provide a better infrastructure for those tools,…

PipeWire: A year in review & a look ahead

08/03/2022

The PipeWire project made major strides over the past few years, bringing shiny new features, and paving the way for new possibilities in…

Landing a new syscall, part 1: What is futex?

08/02/2022

Over the past 18 months, we have been on a roller-coaster ride developing futex2, a new set of system calls. As part of this effort, the…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2022. All rights reserved. Privacy Notice. Sitemap.