Every year, in addition to the Fall conference, the core GStreamer community meets for a Spring hackfest. This year, the community decided to do it right after the libcamera and PipeWire hackfests to foster more coordination between the projects. This completed a strong week in Nice, France, that began with the Linux Media Summit and Embedded Recipes. The hackfest was an occasion to advance various topics that were languishing and to bring the state of the art forward.

The topic of the hour for our team has been the integration of AI/ML workflows in GStreamer. I completed, and finally merged, the plugin to use the LifeRT (formerly TensorFlow Lite) framework, as it is a popular framework for embedded systems. It has been living in a branch for over a year and I'm really happy that it is finally in. One big feature that we added compared to the existing ONNX-Runtime plugin is the addition of a .modelinfo file. It adds metadata not provided by the model. The first part is adding information about the offset and scale (also called mean and standard deviation) to apply to the inputs to the model. But, the most important of those being a name for each tensor, allowing us to prepare the field for auto-plugging the model specific, but framework agnostic, tensor decoders elements. On that topic, Daniel Morin worked hard to improve his new plugin which will automatically select and use the right plugin to decode the tensors into usable information. He's been focusing on the description of the tensors in the caps.

We also had a discussion on the best way to integrate information on tensor pre-processing the caps, this way, such pre-processing could be offloaded to an element before the inference element, for example, combining with a colorspace conversion step. We've decided that we can try to extend the GstVideoInfo API instead of pushing the information outside of the caps.

Bridging AI and PipeWire, Julian Bouzas has been working on Silhouette, an application to create a virtual camera applying a background removal filter to the input from real camera. We've discussed the various challenges faced to integrate GStreamer into a low latency PipeWire pipeline.

Jakub Adam worked on improving DMABuf support within video4linux2 plugin and debugged issues occurring on dynamic caps renegotiation. Turns out proper handling of dynamic pipelines will require more refactoring of how GStreamer deals with V4L2 memory. Jakub and Nicolas Dufresne discussed the needed changes in V4L2 allocator and buffer pool.

On the whole, it was a busy weekend for the community and for Collabora's GStreamer team, and it has already resulted in many thousands of lines of new code landing! With GStreamer 1.27.1 released last week, we are now making our way to the next stable release (1.28), which should be made available by the end of the year. Take a look at this article if you would like to get a glimpse at what's planned for 1.28.