One of GStreamer's key strengths as a multimedia framework is its support for a wide range of hardware and software platforms. GStreamer's upstream Machine Learning support continues this tradition by making it possible to use different frameworks. And we did our first implementation using the equally cross-platform ONNX Runtime, a high-performance inference engine for Open Neural Network Exchange (ONNX) models. ONNX is an open standard format for representing machine learning models that allow models to be trained in one framework and transferred to another for inference. ONNX Runtime will run inference for these models on a growing selection of platforms via its many "execution providers" - supported hardware includes x86 and ARM CPUs, NVIDIA, AMD and Intel GPUs and AMD FPGAs. A full list of supported (upstream) execution providers can be found on the ONNX Runtime website, and many more are available from vendor repositories.

GStreamer has supported ONNX for a number of years via its onnxobjectdetector element, which can detect objects in a video stream using a Single Shot Detector (SSD) model. The CPU and CUDA execution providers are currently supported.

To address a broader range of analytics workflows, Collabora recently redesigned this plugin:

1. All ONNX-related code was moved to a single onnxinference element. Generic inference tensor outputs are now attached to outgoing buffers via the new GstTensorMeta meta-data object and then processed by downstream "tensor decoder" elements.

2. The SSD object detector was redesigned as a much simpler tensor decoder and was moved to the new ssdobjectdetector element.

3. A new objectdetectionoverlay element was added to display analytics meta-data such as detection bounding boxes and associated text labels.

It is now much easier to create new tensor decoders that support many different AI models. It is also now possible to support other inference engines besides ONNX, for example by adding a TensorFlow Lite or a PyTorch inference element.

Here is a high-level view of the new architecture:

Here is how an object detection pipeline might look after this refactor Note: test image file, model file, and label file can be found here:

gst-launch-1.0 multifilesrc location=onnx-models/images/bus.jpg ! jpegdec ! videoconvert ! \
onnxinference execution-provider=cpu model-file=onnx-models/models/ssd_mobilenet_v1_coco.onnx ! \
ssdobjectdetector label-file=onnx-models/labels/COCO_classes.txt  ! \
objectdetectionoverlay object-detection-outline-color=0xFF0000FF draw-labels=true ! \
videoconvert ! autovideosink

Combined with the new analytics meta-data framework, it is now easier than ever to create complex, multi-model GStreamer analytic pipelines that take full advantage of the latest state of the art AI models running efficiently on the latest hardware.

If you would like to find out more about how you can add multi-platform AI analytics to your current GStreamer workflow, please get in touch!

Related Posts

New unixfd plugin in GStreamer 1.24

GStreamer 1.24: New heights in embedded, network streaming & analytics

A framework to share analytics data in GStreamer

Related Posts

New unixfd plugin in GStreamer 1.24

GStreamer 1.24: New heights in embedded, network streaming & analytics

A framework to share analytics data in GStreamer