We're hiring!
*

Every pixel is a measurement: Lossless video compression in GStreamer

Daniel Morin avatar

Daniel Morin
November 18, 2026

Share this post:

Reading time:

zlib, deflate, and Brotli, from pipeline to MP4

Mainstream video codecs such as H.264, HEVC, and AV1 are lossy by design: they discard detail to hit a bitrate, guided by what the human eye is least likely to notice. For geospatial and scientific imagery that trade-off is often unacceptable: a pixel from a multispectral sensor, a thermal camera or a depth map is a measurement, and analysis tools downstream expect to get back exactly the value that was captured.

The obvious answer is to keep the video uncompressed. ISO/IEC 23001-17 already defines how to store uncompressed video (uncv) in ISOBMFF/MP4, and GStreamer supports it. But raw video is big!

Earlier this year, as part of work for the Open Geospatial Consortium (OGC) on GStreamer for geospatial video, I added the missing middle ground to GStreamer: general-purpose lossless compression, usable both as plain pipeline elements and as a standard way to store compressed video in MP4. The elements are not specific to video: they take any media type, so the same approach works for audio, metadata, or any other stream.

All of this is available in the 1.29 development series and will be part of GStreamer 1.30.

Lossless compression elements and MP4 support

  • A new compress plugin in gst-plugins-rs, with three pairs of elements:
    • zlibcompress / zlibdecompress: zlib (deflate with header and checksum)
    • deflatecompress / deflatedecompress: raw deflate
    • brotlicompress / brotlidecompress: Brotli
  • isomp4mux (gst-plugins-rs) can write these compressed streams to MP4 following ISO/IEC 23001-17.
  • qtdemux (gst-plugins-good) can read them back.
  • Both also handle Bayer (video/x-bayer), the raw output of most image sensors, compressed or not.

Because the MP4 files follow ISO/IEC 23001-17, they aren't tied to GStreamer: any tool implementing the standard can read them. Each compressed frame decompresses back to a regular uncompressed frame, and the file still describes its pixel format. A reader that doesn't support the compression recognizes that it can't decode the track, instead of misinterpreting the data.

Caps that remember where they came from

The key design point is caps handling. A compressed buffer is just bytes, but downstream still needs to know what those bytes become once decompressed. The compressor wraps the input caps in an original-caps field:

application/x-zlib-compressed, original-caps=(GstCaps)"video/x-raw, format=NV12, width=3840, height=2160, framerate=30000/1001"

The decompressor unwraps this field and outputs exactly the original caps. No capsfilter is needed after it.

How isomp4mux and qtdemux use original-caps

isomp4mux relies on the same field. The compressed buffers themselves carry no format information, so the muxer reads the width, height, framerate, and pixel format from original-caps and describes them in the MP4 sample entry:

  • The sample entry is a restricted video entry (resv) with the width and height from original-caps.
  • cmpd (component definition) and uncC (uncompressed frame configuration) describe the pixel format: which components there are, their bit depth, the subsampling, and the plane layout.
  • rinf (restricted scheme information) marks the track as generically compressed. Inside it, frma records the original sample entry type (uncv, uncompressed video), schm declares the gcmp scheme, and schi holds a cmpC box naming the algorithm: zlib, defl (deflate), or brot (Brotli).

On the way back, qtdemux reads these boxes and rebuilds the application/x-zlib-compressed caps with original-caps, so the decompressor can output the original pixel format and size.

Using them in a pipeline

Storing a compressed stream with GDP

To store a compressed stream as a plain file, combine the elements with the GStreamer Data Protocol (gdppay/gdpdepay) so caps and events are kept. GDP works with any media type, which makes it the generic option when the data does not fit in a standard container.

Compress first, then wrap with GDP. The compressed buffers are framed and the caps (including original-caps) stay readable without decompressing:

gst-launch-1.0 videotestsrc num-buffers=100 ! zlibcompress ! gdppay ! filesink location=video_zlib.gdp
gst-launch-1.0 filesrc location=video_zlib.gdp ! gdpdepay ! zlibdecompress ! videoconvert ! autovideosink

Wrap with GDP first and everything, including caps and metadata, gets compressed:

gst-launch-1.0 videotestsrc num-buffers=100 ! gdppay ! zlibcompress ! filesink location=video_gdp.zlib
gst-launch-1.0 filesrc location=video_gdp.zlib ! zlibdecompress ! gdpdepay ! videoconvert ! autovideosink

The same pipelines work for audio, and the caps (sample format, rate, channels) travel with the data:

gst-launch-1.0 audiotestsrc num-buffers=300 ! zlibcompress ! gdppay ! filesink location=audio_zlib.gdp
gst-launch-1.0 filesrc location=audio_zlib.gdp ! gdpdepay ! zlibdecompress ! audioconvert ! autoaudiosink

The full round trip with MP4

This is where the standard comes in. Compress raw video and mux it into MP4:

gst-launch-1.0 videotestsrc num-buffers=60 ! video/x-raw,format=RGB,width=640,height=480,framerate=30/1 ! \
    zlibcompress ! isomp4mux ! filesink location=video_zlib.mp4

Read it back with qtdemux. It exposes application/x-zlib-compressed caps with the original-caps reconstructed from the uncC/cmpd boxes, so the decompressor knows what it is producing:

gst-launch-1.0 filesrc location=video_zlib.mp4 ! qtdemux ! zlibdecompress ! videoconvert ! autovideosink

Replace zlib with deflate or brotli and everything else stays the same.

Raw sensor data: Bayer

Most image sensors do not output RGB. They output a Bayer mosaic, one color sample per pixel, and the color image is only reconstructed later. For scientific imagery that matters: the Bayer data is the closest thing to the original measurement, and converting it to RGB already changes the values. isomp4mux and qtdemux can store and read Bayer data directly, with or without compression.

A sensor that exposes its raw Bayer output can be captured with v4l2src, which produces video/x-bayer caps, and written straight to MP4:

gst-launch-1.0 -e v4l2src device=/dev/video0 ! zlibcompress ! isomp4mux ! filesink location=bayer_zlib.mp4

Without a sensor at hand, the rest of this section simulates the capture: rgb2bayer turns a test pattern into a Bayer stream, and the output file is the same kind of MP4.

gst-launch-1.0 videotestsrc num-buffers=60 ! video/x-raw,format=ARGB,width=640,height=480,framerate=30/1 ! \
    rgb2bayer ! video/x-bayer,format=rggb ! zlibcompress ! isomp4mux ! filesink location=bayer_zlib.mp4

The MP4 describes the Bayer data itself. Reading it back without any caps filter, the decompressor outputs the exact format that went in (printed here with -v):

gst-launch-1.0 -v filesrc location=bayer_zlib.mp4 ! qtdemux ! zlibdecompress ! fakesink
...
video/x-bayer, format=(string)rggb, width=(int)640, height=(int)480

To look at the frames, bayer2rgb demosaics them for display:

gst-launch-1.0 filesrc location=bayer_zlib.mp4 ! qtdemux ! zlibdecompress ! bayer2rgb ! video/x-raw,format=BGRx ! videoconvert ! autovideosink

Drop zlibcompress and zlibdecompress to store the Bayer data uncompressed. All the Bayer patterns (rggb, bggr, grbg, gbrg) and bit depths up to 16 bits are supported.

MP4 or GDP?

Compressed data can be stored either with isomp4mux or with gdppay. The compressed data is the same in both cases, so the choice comes down to two questions: what is being compressed, and who needs to read the result.

  • What is compressed. MP4 can only hold what ISO/IEC 23001-17 can describe: uncompressed video formats (such as RGB, I420, or NV12) and Bayer. Audio, metadata, or any other stream cannot be written as a generically compressed MP4 track. GDP has no such limit: it carries any media type, with its caps.
  • Interoperability. An MP4 follows a standard, so any tool implementing ISO/IEC 23001-17 can read it. A GDP file can only be read by GStreamer.

Use MP4 for video you want to share or archive in a standard format, and GDP for everything else, or when the data stays within GStreamer applications.

Trying it on real footage

Test patterns are fine for showing the syntax, but I also wanted to show the elements on something closer to a real geospatial workload: drone footage of a forested area, 4K at 30 fps.

Aerial view of a forest edge next to a field

First frame of the footage, scaled down to 1080p and saved as JPEG for this page. Footage: Serene Aerial View of Dense Coniferous Forest by SALAR ♛ on Pexels.

The raw video capture is not publicly available, so I decoded the MP4 from Pexels to NV12 (the pixel format most capture hardware delivers) and used the headerless raw file as a simulated capture source:

gst-launch-1.0 filesrc location=footage.mp4 ! decodebin3 ! videoconvert ! \
    video/x-raw,format=NV12 ! filesink location=footage_4k_nv12_30fps.raw

The pipelines

A raw file carries no caps, so the capture is simulated by injecting them with rawvideoparse, which also timestamps every frame at the right rate. The source then goes through compression and MP4 muxing, and the result is demuxed and decompressed back to video:

gst-launch-1.0 filesrc location=footage_4k_nv12_30fps.raw ! \
    rawvideoparse format=nv12 width=3840 height=2160 framerate=30000/1001 ! \
    zlibcompress level=6 ! isomp4mux ! filesink location=footage_zlib_6.mp4

gst-launch-1.0 filesrc location=footage_zlib_6.mp4 ! qtdemux ! zlibdecompress ! videoconvert ! glimagesink

I ran this for every codec at four levels: 0, 1, 6, and the maximum (9 for zlib and deflate, 11 for Brotli). Every file decompressed back to frames identical to the source, checked with the compare element.

Results

Codec Level Ratio (raw ÷ compressed) Max compress fps Max decompress fps
zlib 0 1.00 625 360
zlib 1 1.23 17.3 42.9
zlib 6 1.63 6.5 45.6
zlib 9 1.67 3.5 36.9
deflate 0 1.00 909 414
deflate 1 1.23 17.2 43.5
deflate 6 1.63 6.6 45.9
deflate 9 1.67 3.5 37.3
Brotli 0 1.40 16.4 21.8
Brotli 1 1.42 14.5 20.2
Brotli 6 1.66 1.7 20.6
Brotli 11 1.91 0.075 19.7


The frame rates are the maximum one stream can sustain through each element alone, for 4K NV12 frames on one CPU core (AMD Ryzen AI 7 350).

Brotli compresses best, up to 1.91 at level 11, but at that level it takes over 13 seconds per frame, which makes it a setting for archiving rather than capture. At levels 0 and 1 it compresses at about the same speed as zlib at level 1, with a noticeably better ratio, but it decompresses at half the speed. zlib and deflate give the same sizes and speeds, since deflate is the same algorithm without the zlib header and checksum. Decompression is much cheaper than compression, and its cost barely depends on the level. No level compresses 4K at 30 fps on a single core: the fastest real compression, zlib or deflate at level 1, reaches about 17 fps. The elements process one frame at a time, so this is a per-stream figure.

Live playback

To see whether playback keeps up, I played each file on screen with fpsdisplaysink, which counts rendered and dropped frames:

gst-launch-1.0 -v filesrc location=footage_zlib_6.mp4 ! qtdemux ! queue ! zlibdecompress ! queue ! \
    fpsdisplaysink video-sink=glimagesink text-overlay=false sync=true

zlib and deflate played the 4K footage at 29.97 fps without dropping a single frame, at every level, the same as the raw footage without compression. Brotli needs about 50 ms per frame, but the capture timestamps leave only 33 ms between frames, so the video sink drops nearly all of them. A workaround, at the cost of temporal resolution, is to keep every other frame with videorate: Brotli then played live without dropping a frame.

The bottleneck is decompression. The disk reads these files at 4.4 GB/s, while live playback of the level 6 file needs about 230 MB/s, and the display handled the raw 4K footage (about 373 MB/s) at full rate.

Measuring it yourself

The per-frame times come from the latency tracer, which logs how long each element holds every buffer. Name the element you want to measure, run the pipeline with the tracer, and average its entries:

GST_TRACERS='latency(flags=element)' GST_DEBUG='GST_TRACER:7' GST_DEBUG_NO_COLOR=1 GST_DEBUG_FILE=trace.log \
gst-launch-1.0 filesrc location=footage_4k_nv12_30fps.raw ! \
    rawvideoparse format=nv12 width=3840 height=2160 framerate=30000/1001 ! \
    zlibcompress level=6 name=c ! isomp4mux ! filesink location=footage_zlib_6.mp4

The log has one element-latency line per frame, with the time the element held it in nanoseconds. This small script averages them for one element:

python
import re
import sys

log_file, element = sys.argv[1], sys.argv[2]

times = []
with open(log_file) as f:
    for line in f:
        match = re.search(rf"element=\(string\){element}, .*time=\(guint64\)(\d+)", line)
        if match:
            times.append(int(match.group(1)))

avg_ms = sum(times) / len(times) / 1e6
print(f"{element}: {len(times)} frames, {avg_ms:.1f} ms/frame, {1000 / avg_ms:.1f} fps")
$ python3 element_fps.py trace.log c
c: 686 frames, 154.1 ms/frame, 6.5 fps

For decompression, name the decompressor instead and send its output to fakesink sync=false, so it runs as fast as it can.

Conclusion

One hour of raw 4K NV12 at 30 fps takes 1.34 TB. With zlib or deflate at level 6, the same hour takes 824 GB, and Brotli at level 11 brings it down to 703 GB, with every frame decompressing back to exactly what was captured.

Lossy codecs go much further. The H.264 file the footage came from is about 29 times smaller than the raw video, while lossless compression makes the raw video 1.6 to 1.9 times smaller at level 6 and above. But for geospatial data, every pixel is a measurement, and analysis tools must get back exactly the value the sensor captured, which a lossy codec cannot guarantee.

A natural next improvement is parallelism. The elements currently process one frame at a time on a single core, which is why compression falls short of a 4K capture here, and Brotli of real-time decompression. Since every frame is compressed independently, a future version could compress several frames in parallel on separate cores, at the cost of a little latency: roughly the time it takes to process one frame. In principle, five cores would be enough for zlib at level 6 to compress 4K at 30 fps, and two for Brotli to decompress it in real time.

Try it out

The compress elements and the MP4 support are available in the GStreamer 1.29 development series and will ship in GStreamer 1.30. To try them now, build GStreamer from the main branch with the Rust plugins enabled. This needs a Rust toolchain (currently 1.92 or newer) and cargo-c (cargo install cargo-c):

git clone https://gitlab.freedesktop.org/gstreamer/gstreamer.git
cd gstreamer
meson setup builddir -Drs=enabled
meson compile -C builddir
meson devenv -C builddir

The last command opens a shell where gst-launch-1.0 uses the freshly built plugins.

If you work with sensor or geospatial video and try this out, I'd love to hear how it works for your data: which compression ratios you get, which pixel formats you need, and what’s missing. Bug reports and feature requests are welcome on the GStreamer GitLab.

And of course, if you're building a geospatial or scientific video pipeline and need help with GStreamer, get in touch!

Search the newsroom

Latest Blog Posts

Every pixel is a measurement: Lossless video compression in GStreamer

18/11/2026

Lossy codecs discard data that geospatial analysis needs. GStreamer now adds lossless zlib, deflate, and Brotli compression for raw and…

Sunscreen, a stylus, and 16 hours of pair programming with an AI

02/11/2026

An AI-assisted side project turned a missing KDE tablet feature into a working handwriting mode, combining open source recognition, stylus…

Upstreaming Tyr: Rust, DRM, and the Linux Kernel

21/10/2026

Tyr has reached a major milestone by booting the Arm Mali-G610 GPU’s MCU using upstream Rust DRM memory-management abstractions, while helping…

Building Tyr in Rust, part 3: Managing GPU memory

26/08/2026

Tyr’s latest Rust GPU kernel driver work puts GPU memory management in place, from VM binding and GPUVM to GEM-backed buffers and firmware…

The power of APIs: The unsung hero of AI interface

07/07/2026

AI development is shifting from implementing models from scratch to composing powerful capabilities via APIs, enabling developers to integrate…

Simplifying Bluetooth qualification for Linux/BlueZ: New upstream documentation

26/05/2026

New upstream BlueZ documentation helps simplify Bluetooth qualification for Linux-based products by mapping supported profiles, test requirements,…

Open Since 2005 logo

Our website only uses a strictly necessary session cookie provided by our CMS system. To find out more please follow this link.

Collabora Limited © 2005-2026. All rights reserved. Privacy Notice. Sitemap.