This is the third and last part of a series explaining how Collabora is helping shape the video virtualization story for Chromebooks. Part one looked at hardware acceleration for video codecs and the VirtIO Video protocol. Part two explored the Rust libraries Collabora wrote to decode video content.

This final installment discusses the future plans for cros-libva and cros-codecs in general, as well as the tentative "virtio-media" protocol and some of our recent contributions to the ecosystem.

State-of-the-art codec support

As we have previously mentioned, cros-codecs used to only support VP8, VP9, and H.264 decoding. This made it a formidable video decoding library, but the world has since moved on to newer and more exciting codecs. HEVC has found wide support on Apple devices and on some streaming platforms. At the same time, AV1 has revolutionized the world of codecs by providing a way forward from the incessant patent litigation of the past while providing world-class compression performance at acceptable bitrates.

It is no surprise to see AV1 being adopted at a fast pace on YouTube. In fact, YouTube plans to revolutionize streaming by pushing AV1 live-streaming support to their platform, and this could not come at a better time, as OBS Studio recently rolled out AV1 live-streaming support.

On the hardware side, we are starting to see support roll out among the big players. Intel has offered AV1 hardware decode from Tiger Lake onwards, AMD from the RDNA2 architecture onwards, and NVIDIA from RTX 3000 GPUs onwards.

As everybody gears up for the latest and greatest in the codec world, we have merged support for both AV1 and HEVC codecs into the cros-codecs library. This means that AV1 and HEVC are now supported for Android applications running on Chrome OS devices, taking advantage of the ARCVM virtualization framework and bypassing the dependency on the Chrome video decoding stack. It also means that we can extend and test the VirtIO Video protocol with HEVC and AV1 directly.

Our efforts on zero-copy video decoding

The problem of one-too-many-copies is definitely not unheard of in software engineering. The extra copying of data to and from the codec's hardware memory used to be a bottleneck when using cros-codecs, and it showed in the form of lower than expected decoding speeds - in particular for 4K content.

We worked together with the ChromeOS team to solve this by relying on DMA-BUF to cheaply import the host's GPU memory into the guest's VA-API context. This lets VA-API render on the target memory directly, eliminating one copy. It massively improved the FPS count on 4k content, allowing us to decode 4K60P video without a hitch. It should also have a pretty noticeable effect on power consumption as well.

A new virtIO protocol

Much has been said here about VirtIO Video, a proposed standard to access video encoders and decoders in VMs, and about how it aims to improve the video codec virtualization landscape, in particular for Chromebooks and ARCVM. But in the last few years, the idea of VirtIO Video itself has been challenged.

The idea for a new protocol spawns from a couple of realizations: first, both VirtIO Video and the tentative VirtIO Camera - a prototype which was put forward by Collabora’s Dmitry Osipenko - heavily resemble the V4L2 uAPI already. In other words, there is room for refactoring such common origins into a single protocol. Second, these protocols - and any other future protocol aimed at virtualizing a new V4L2 device - would represent yet another heavy layer that would itself have to be kept in sync with the V4L2 API.

A separate strategy, being prototyped by Alex Courbot from ChromeOS, is to simply rewrite the VirtIO Video protocol so as to virtualize V4L2 ioctls directly, with no translation in between. He has named it "virtio-media" and the discussion is ongoing.

This scenario would see the VirtIO technology used as a simple transport for the V4L2 kernel-userspace API. This would lead to a smaller VirtIO device specification and kernel driver and it would bring support to cameras, image processors, and other V4L2 devices under the same umbrella, reducing the number of different VirtIO protocols out there. It also benefits directly from the established V4L2 specifications and from the large collection of working V4L2 examples.

In any case, the design of virtio-media and whether it is to eventually supersede VirtIO Video is very much an open question, with industry players actively debating over the pros and cons of each approach.

Going forward: automated testing

Testing the different components of the stack is paramount to ensure their robustness and to assert them as suitable for production use.

Cros-codecs and cros-libva fortunately come with plenty of unit tests that can be run by CI to ensure that no regressions take place, while end-to-end testing of the entire decoding stack (including the virtio-video layer) can be done through Fluster.

Fluster is a testing framework for decoder compliance. It has proven itself as a reliable way to identify regressions in decoders by comparing the results with a set of references, usually produced by the official decoder model provided by the codec standard in use. One can use Fluster to run a decoding loop on the guest. By doing so, they will be testing all components in the stack, including those on the host!

Additionally, Fluster is able to detect deadlocks that would otherwise cause the decode operation to timeout, potentially locking up the hardware until the next reboot, which is why it is important to invest in a plan to run it in an automated manner.

The VISL virtual V4L2 driver is also a welcome addition as a testing tool for cros-codecs once a V4L2 stateless backend is merged. Written by Collabora, VISL is a mock driver that implements the V4L2 stateless APIs. It can be used to test the application code as well as the V4L2 framework inside the Linux kernel without having to rely on a specific hardware platform. This makes it possible to run tests in a generic cloud instance.

Going forward: video post-processing in hardware

It is not uncommon for video decoding hardware to expose post-processing blocks that can execute post-processing algorithms like scaling and denoising in a fast and power-efficient way. Some video decoding APIs, like VA-API, provide a way to interact with these hardware blocks, but no support has been added to cros-codecs yet. This is something that can be explored in a future iteration of the library.

What is now being worked on?

As the cros-codecs project matures, a new area is being tackled by ChromeOS: that of video encoders. On this front, a new encoding interface is being worked on to allow hardware-accelerated video encoding on ARCVM. Another major win for users who may find themselves encoding media for the purposes of video calls or general video content creation. This will initially use VA-API for hardware acceleration, but other backends may be added in the future.

Speaking of other backends and APIs, the ChromeOS team is also working on a stateful interface for video decoding. As Alex Courbot puts it:

The first use of the stateful decoder interface should be as a stateful wrapper to any stateless decoder, i.e. an adapter that turns a stateless decoder into a stateful one.

This will, among other things, make it easier to implement stateful video backends in the future. In particular, this will pave the way to V4L2 stateful support, which will broaden the amount of hardware cros-codecs is able to run on.

Another interesting development is using some of the cros-codecs parsers in GStreamer as a way to crush down on the growing number of CVEs. Bitstream parsers are hard to write and prone to vulnerabilities and thus stand to benefit greatly from the Rust programming language. How exactly this integration between cros-codecs and GStreamer will happen remains to be seen, but the discussion is ongoing.

Wrapping it up

Over the past few years, Collabora has been hard at work with the ChromeOS engineers to improve the video codec stack on Chromebooks. As a result, a few projects have been released as standalone codebases to the open source community, which is a great contribution to the ecosystem in general.

We have also contributed to CrosVM and VirtIO Video: by providing a VA-API based VirtIO Video backend in CrosVM we have provided a way to develop the VirtIO Video protocol on regular workstations. In addition, this has created a path forward to deprecate the use of libvda, which tied CrosVM and the Chrome browser together.

We hope that our multimedia expertise helps propel either the VirtIO Video protocol itself, as it can be used by different industry members outside of a ChromeOS context, or a spinoff aimed at codecs specifically. But most importantly, we also look forward to the concrete benefits that our work will bring to ChromeOS users worldwide.