We're hiring!

Effortless GStreamer Analytics Cross-Platform Support via ONNX Runtime

Aaron Boxer avatar

Aaron Boxer
April 29, 2024

Share this post:

Reading time:

One of GStreamer's key strengths as a multimedia framework is its support for a wide range of hardware and software platforms. GStreamer's upstream Machine Learning support continues this tradition by making it possible to use different frameworks. And we did our first implementation using the equally cross-platform ONNX Runtime, a high-performance inference engine for Open Neural Network Exchange (ONNX) models. ONNX is an open standard format for representing machine learning models that allow models to be trained in one framework and transferred to another for inference. ONNX Runtime will run inference for these models on a growing selection of platforms via its many "execution providers" - supported hardware includes x86 and ARM CPUs, NVIDIA, AMD and Intel GPUs and AMD FPGAs. A full list of supported (upstream) execution providers can be found on the ONNX Runtime website, and many more are available from vendor repositories.

GStreamer has supported ONNX for a number of years via its onnxobjectdetector element, which can detect objects in a video stream using a Single Shot Detector (SSD) model. The CPU and CUDA execution providers are currently supported.

To address a broader range of analytics workflows, Collabora recently redesigned this plugin:

1. All ONNX-related code was moved to a single onnxinference element. Generic inference tensor outputs are now attached to outgoing buffers via the new GstTensorMeta meta-data object and then processed by downstream "tensor decoder" elements.

2. The SSD object detector was redesigned as a much simpler tensor decoder and was moved to the new ssdobjectdetector element.

3. A new objectdetectionoverlay element was added to display analytics meta-data such as detection bounding boxes and associated text labels.

It is now much easier to create new tensor decoders that support many different AI models. It is also now possible to support other inference engines besides ONNX, for example by adding a TensorFlow Lite or a PyTorch inference element.

Here is a high-level view of the new architecture:

Here is how an object detection pipeline might look after this refactor Note: test image file, model file, and label file can be found here:

gst-launch-1.0 multifilesrc location=onnx-models/images/bus.jpg ! jpegdec ! videoconvert ! \
onnxinference execution-provider=cpu model-file=onnx-models/models/ssd_mobilenet_v1_coco.onnx ! \
ssdobjectdetector label-file=onnx-models/labels/COCO_classes.txt  ! \
objectdetectionoverlay object-detection-outline-color=0xFF0000FF draw-labels=true ! \
videoconvert ! autovideosink

Collabora - GStreamer SSD object detector

Combined with the new analytics meta-data framework, it is now easier than ever to create complex, multi-model GStreamer analytic pipelines that take full advantage of the latest state of the art AI models running efficiently on the latest hardware.

If you would like to find out more about how you can add multi-platform AI analytics to your current GStreamer workflow, please get in touch!


Comments (0)

Add a Comment

Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest News & Events

Mesa 24.1 brings new hardware support for Arm and NVIDIA GPUs


Mesa 24.1 is out, and the graphics team at Collabora has been working hard to bring a slew of new features and bug fixes to Panfrost, Zink,…

Implementing DRM format modifiers in NVK


This week we merged support for the VK_EXT_image_drm_format_modifier extension in NVK, the new open-source Vulkan driver for NVIDIA hardware.…

Kernel 6.9: Enable, test, repeat


Collabora's engineers continue to be involved in the hardware enablement for a few different system-on-chips (SoCs) and platforms, and have…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2024. All rights reserved. Privacy Notice. Sitemap.