Next-generation Graphics

One of the most difficult challenges in creating any polished and smooth system—from an embedded system running your car stereo to a full-fledged desktop—is getting the graphics software right.

The graphics stack does not simply stop at having OpenGL ES-capable hardware. As user requirements have grown and evolved beyond having a single full-screen game, so have the demands on the graphics software, to the point where it is one of the most complex — yet least well-understood — parts of a modern system. The graphics pipeline can no more be reduced to "display some content" than media can be reduced to "play a video": it requires a careful balancing act between visible performance on one hand, and power and memory consumption on the other, whilst ensuring proper synchronisation, fluid animations, high throughput, accurate colour reproduction, and more.

Working across a myriad of form factors and device types also presents a strong challenge. A change which improves performance for a particular usecase on some hardware may also, counterintuitively, cause a severe performance decrease on another device target.

Wayland, the next-generation window system

X11 has been the industry standard for almost all UNIX systems since its inception in 1986. Collabora engineers have been heavily involved in advancing the development of X11 for over a decade between them, but its fundamental limitations have led the open source community to develop a replacement more suited to modern and fluid UIs.

Wayland is an open source initiative backed by Intel, Red Hat, Collabora and more, which aims to produce a base on which applications can deliver fluid, modern user interfaces. Wayland acts as a replacement for X11, integrating with OpenGL ES and EGL for 3D-rendered display, media frameworks such as GStreamer, OpenMAX and VA-API for video playback, and more. With further support from the X.Org development community, semiconductor vendors, and OEMs, the Wayland project is paving the way for the future of open-source graphics.

Wayland's core underlying principle is simple: every frame must be perfect. By that, Collabora means that the user must never see incomplete or transitional results: seeing a window appear first as a gray square, followed by half of the UI elements, with the rest filled in at a later date, is simply not good enough by modern standards. Similarly, users are now unwilling to accept jerky animations, seemingly random freezes, or anything less than perfect graphical interfaces. While X11 must be fought to achieve these results, often with band-aid solutions which are not portable beyond a single product, Wayland has these principles baked in from scratch.

Another core goal of Wayland is to vastly improve the driver support story compared to X11. The X11 driver model is extremely complex and fragile, and requires extensive integration from the hardware vendor in order to deliver basic enablement, as well as ongoing rolling updates whenever the overall software stack is updated. Wayland offers a much smaller surface area, requiring only support for standard Linux kernel display APIs, as well as one EGL extension for accelerated 3D. This not only decreases the upfront development cost to the hardware vendor, but drastically reduces the ongoing support cost.

The Wayland project is comprised of the following:

  • The core Wayland display protocol, defining how compositors (display servers) interact with clients (including other display servers for compatibility purposes, such as X.Org), as well as input and output devices;
  • A reference library implementing the Wayland protocol, offering a native C API, and supporting binding to other languages; and
  • Weston, the reference implementation of a fully-featured compositor.

Collabora works closely on the entire Wayland ecosystem, from core protocol changes affecting how all content is displayed on screen, to new features inside Weston's UI, to hardware enablement on a number of platforms, to adding and enhancing Wayland support for higher-level toolkits such as Clutter and GTK+. Collabora's deep involvement, across a number of devices, applications and verticals, gives Collabora a unique insight into Wayland's design and direction.

Weston: next-generation compositing and window management

Weston is a compositor implementation targeted as the base for embedded, mobile and desktop environments. Weston provides three major components: the core Wayland protocol handling, an abstracted hardware layer allowing it to run on a number of platforms with varied hardware and display control interfaces, and a UI framework allowing rapid development of new interfaces. Weston supports the most common input and output frameworks under the Linux kernel, and is able to use any of OpenGL ES, Pixman, or dedicated display hardware (see the Raspberry Pi case studyfor more information). Its first-class support for running inside an existing X11 or Wayland session allows rapid development and prototyping without the need for a separate virtual machine.

Perhaps Weston's most compelling feature for embedded, mobile and media platforms is its extensive support for dedicated display hardware. Most X11-based systems either do not support use of dedicated 2D compositing blocks, requiring either power-hungry OpenGL or slow software-based rendering, or suffer from a lack of integration between the hardware acceleration and user interface. Weston supports dedicated compositing hardware as a first-class citizen, enabling fluid animations and transitions and high-quality video decoding, all whilst keeping power usage and CPU utilisation to an absolute minimum. With this, Collabora is able to deliver slick user interfaces to platforms whose true hardware capabilities have often gone to waste.

Drivers and hardware enablement

Collabora has successfully delivered Wayland hardware enablement for a number of platforms, from powerful media-led hardware to mobile devices and everything in between. For instance, Collabora contributed to Wayland-based solutions for the Raspberry Pi, Android, PowerVR RGX-based SOCs, etc.

Wayland delivers a much lower initial and ongoing cost with regards to hardware enablement than previous heavyweight solutions. Collabora is able to rapidly deliver a flexible solution which meets any needs.

One key feature of Collabora's hardware enablement has been a fully zero-copy pipeline, allowing our customers to achieve the highest possible throughput, which is particularly crucial for media and video applications, as well as high-resolution and high-DPI displays. Wayland's unique architecture has allowed Collabora to include platform-specific optimisations and tweaks crucial for full performance without invasive patches to Wayland itself, allowing full portability between Wayland implementations whilst still using the hardware to its full potential.

Graphic toolkits for rich user interfaces

Most applications are written using a toolkit such as GTK+, Qt or Clutter, which provides the basic UI elements and look and feel for applications, as well as papering over differences in the underlying graphics stacks.

With many years of experience on the desktop and mobile stacks, be it for Linux or other platforms, Collabora is uniquely positioned to design and implement beautiful and functional graphical user interfaces. Collabora uses technologies such as GTK+, Qt, Clutter, Mx, and WebKit on a regular basis. Using those toolkits is not all, of course: Collabora has also been helping adapt those toolkits, along with various applications and frameworks, so that they work with Wayland.

Should your needs go beyond what these toolkits already offer, Collabora can help you improve them and contribute the improvements back to the community.

Collabora also offers ongoing support, training around these technologies or user interaction design services.

How Collabora can help you with the graphics software stack

  • Porting and optimization to your platform(s) and architecture(s) (ex: BSD compatibility, ARM-based graphics-intensive devices such as televisions, digital signage, smartphones, interactive cameras, etc.
  • Hardware enablement: Collabora can help you improve your EGL stack to support Wayland
  • Should you need extensions to the Wayland protocol to address problems you might have, Collabora can help you design, draft and integrate such extensions for the Wayland project. For example, if you need to interface your own display API with Wayland, Collabora can help you write a new backend to Weston.
  • Collabora can help you in the transition of your applications from X11/X.org to Wayland.
  • Collabora can further assist with performance optimization for XWayland, the compatibility layer for X11 applications, given a native DDX for a particular piece of hardware you may wish to use.
  • Improvements to widget and animation toolkits such as GTK+, Clutter, Webkit, Qt, etc.
  • Collabora can implement a zero-copy, hardware accelerated video presentation path on top of proprietary video decoding, processing, and presentation APIs, provided that they are technically compatible with Wayland.
  • Collabora can integrate GStreamer hardware-accelerated video decoders into Wayland, allowing zero-copy media playback and support for rich media functionality in complex applications such as GLES composition in WebKit for HTML5 video.

...and more! Please get in touch with us so we can start helping you today.

Open Source Projects

Click below to learn more about the various Open Source projects we're actively involved in.

Linux kernel GStreamer PipeWire Wayland WebKit Debian PulseAudio Rygel libnice D-Bus BlueZ LibreOffice
Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2024. All rights reserved. Privacy Notice. Sitemap.