December 20, 2021
From its inception, an important use case for xrdesktop was that applications that are already running on the 2D desktop should be seamlessly available in VR at the press of a button. Although it is possible to implement something like this in a desktop agnostic fashion with pure xlib/xcb proof of concept: x3d, for performance and robustness reasons we opted for code integration into window managers. Typically, window managers have an internal abstraction of the windows they manage that is much easier to reliably consume than the raw X11 protocol.
The first two window manager integrations we started out with were a patchset for gnome-shell and a kwin plugin. The window manager integration approach unfortunately suffers several drawbacks.
Most importantly however, not everyone is running kwin or gnome-shell, and adding integration into more and more window managers would result in an unmaintainable mess.
Each window manager is shipped in a different version on different distributions too. The gnome-shell patchset needs to be adjusted for every major version change. The kwin plugin needs to be compiled for a specific kwin version, meaning an Ubuntu PPA with an updated KDE version would break a packaged version of the plugin.
While the ability to mirror already running 2D applications without good performance has been a unique feature of xrdesktop, which is enjoyed by our users and will still be maintained, we have also heard many voices who desire a more native XR solution, especially looking at standalone devices.
As an alternative to the window manager integration, we developed a standalone client based on wlroots and the wxrc compositor which had been dormant until recently. wxrd follows a similar approach as Simula VR (haskell, godot) or Stardust (stereokit), where the XR desktop is a standalone application that will not mirror existing windows from a 2D desktop, but only show windows in XR that have been explicitly started on this XR desktop. Other noteworthy pioneers are motorcar, a kwayland based VR Wayland compositor, and safespaces, an entire custom Wayland and X11 implementation. What sets xrdesktop apart from these alternatives is that xrdesktop's tech stack is focused on a small footprint. Instead of depending on large frameworks and libraries, xrdesktop with wxrd focuses on minimal dependencies and a manageable scope.
The wxrd standalone client is implemented as a Wayland compositor, but that doesn't mean you have to run your desktop on Wayland to use it. Wlroots supports various backends, for example its X11 backend allows running wlroots based compositors in a X11 desktop window.
Thanks to wlroots' xwayland implementation, not only can wxrd display Wayland applications in VR, but also X11 applications.
There are major advantages of running a standalone compositor over a 2D window manager integration:
WAYLAND_DISPLAY=wayland-0), the windows already running on your 2D desktop won't be mirrored;
Part 1 shows how xwayland seamlessly allows X11 applications to run. glxgears is running at 144 frames per second, the native refresh rate of the Valve Index used in the recording of this video.
Part 2 shows xrgears (another OpenXR application) being launched from wxrd.
Part 3 demonstrates how both a physical keyboard and the VR keyboard can be used to input text into applications. Even special unicode characters like Emojis work.
Part 4 demonstrates how the window focus for keyboard input can be changed with just the physical keyboard without the use of VR controllers. This lays the technical groundwork for UX with keyboard and mouse in addition to VR controllers.
The Wayland core protocol specifies keyboard input via keymaps and keycodes. A keymap describes the layout of a physical keyboard, including which characters are available on that keyboard. The keycode then describes specifically for this layout which key is pressed.
Advanced text input methods are being added to Wayland with a text-input and input-method protocol. These input methods support sending unicode characters like emojis and strings directly to applications, and allow native implementations of rich predictive text input. Input methods unfortunately have the drawback that Wayland clients need to support the protocol to be able to receive text.
Furthermore, a virtual keyboard protocol is actively being developed with an interest on being able to provide input with a virtual keyboard for all applications. Unfortunately this work is not finished yet.
Therefore wxrd starts with a basic implementation of emulating keyboard input with keymaps created by the xkbcommon library and keycodes. Gamescope and Phosh's squeekboard implement this mode too.
Because keymaps aren't large enough to hold all possible unicode characters, every time the compositor wants to send text to a client, it dynamically creates a new keymap that contains exactly the characters contained in the string, sets this keymap as the active keymap, then sends the corresponding key codes.
With this technique, wxrd can process all unicode characters including emojis from xrdesktop's VR keyboard.
In the future we hope to use the advanced input method protocols for experimenting with new text input methods that can only be achieved in VR.
In wlroots the main purposes of a renderer are informing applications about available/supported formats, and when applications submit a rendered frame in either a pixel buffer in system ram or a file descriptor pointing at GPU memory, the renderer imports this memory so the compositor can present them in some composited way.
In the wlr_renderer interface header the functions a renderer has to implement can be seen. The most important ones are:
wlroots 0.14 creates an internal gles2 based renderer with wlr_backend_autocreate(). This was changed in wlroots master (0.15) and a custom renderer can completely replace the internal default renderer.
wxrd implements a custom renderer called wxrd_renderer based on xrdesktop's gulkan library. Another project that implements a custom Vulkan renderer for wlroots is gamescope.
A native Vulkan renderer in wlroots has been contributed in wlroots master (0.15) but it is not yet ready for some of the usage required by wxrd: The wlroots Vulkan renderer will internally initialize a VkInstance and VkDevice, whereas xrdesktop internally must use the VkInstance and VkDevice that has been handed down from the OpenXR runtime. Future functionality like "Add a function to get a VkImage from a wlr_texture" from issue 3267 will help with sharing Vulkan textures between the Vulkan instances from wlroots and xrdesktop. Eventually we will be able to drop most of our own wxrd_renderer implementation in favor of wlroots' Vulkan native renderer.
The X11 and Wayland backends provided by wlroots work fine for users who already run a 2D desktop session on X11 or Wayland. On standalone devices it is desirable to not run such a desktop session at all.
Unfortunately when we try to run wxrd without an X server or Wayland compositor "on drm", we run into the problem that in this setup we also want to run a VR compositor on drm. Both the wlroots drm backend and a VR compositor require to be drm master to run.
"drm master" is a permission that only one process at a time can acquire, which enables this process to issue ioctls that for example control the display resolution and refresh rate. Usually this process is the X.org server or a Wayland compositor. Many embedded devices that only need to display a single application don't bother running such a "heavy" graphical session and implement rendering their application directly on drm, requiring the drm master permission. Unfortunately the wlroots drm backend and the monado compositor are both such applications.
wlroots has a solution to this, the headless backend. wlroots 0.14 also provided a noop backends that compositors with custom renderers like gamescope preferred because the noop backend was more lightweight, for example it was the only backend not creating a renderer by default. Now in wlroots master (0.15) the headless backend became so similar to the noop backend, that the noop backend was dropped.
wxrd uses the headless backend when it detects being run on drm or when the environment variable WXRD_HEADLESS=1 is used on X11 or Wayland. When wxrd is started in its default mode in X11 or Wayland, it creates an empty window that is used to grab physical keyboard and mouse input. In headless mode wxrd does not create a "window" at all.
This means that currently headless mode only processes input from the VR controller and VR keyboard. In the future, wxrd can solve this by dynamically opening a window like gamescope to grab physical mouse and keyboard input on X11 and Wayland, and on drm wxrd can attach a "libinput" backend provided by wlroots. Prior art for these implementations exists in gamescope too.
An unexpected obstacle in the implementation of wxrd was the state of importing dma-buf based Wayland buffers into Vulkan. Importing a block of GPU memory from a dma-buf file descriptor into Vulkan is easy in theory and something xrdesktop's gulkan library was already capable of doing - as long as the layout of the GPU memory is "linear" and "not tiled".
Modern GPUs and drivers typically store texture data in hardware specific ways, including hardware specific compression and tiling, which helps with bandwidth and GPU memory usage in general. For typical users of graphics APIs this is entirely transparent but when sharing a reference to a certain block of GPU memory between two different APIs or application processes, the receiving end typically has to be told with what settings this block of memory is expected to be used. Thus drm format modifiers and an accompanying Vulkan extension VK_EXT_image_drm_format_modifier were created.
"A drm format modifier is one 64-bit number that has to be passed from the exporter to the importer" - Sounds simple, right? Not quite, as some drm format modifiers specify a storage format that uses multiple memory planes. A buffer could consist of multiple memory planes for two reasons:
One property of the second case is that while each plane is represented by a separate dma-buf, both of these dma-bufs will point to the same memory location and the second plane will be signified by an offset. The Vulkan specification currently does not document well how this case must be handled.
It is clear is that when first creating the VkImage that will be used to import the buffer, a VkSubresourceLayout struct must be chained to VkImageCreateInfo::pNext, containing one VkSubresourceLayout struct with metadata like size and memory offset per each memory plane.
The explicit mechanism for binding multiple memory planes to one VkImage is by calling vkBindImageMemory2 with an array of multiple VkBindImagePlaneMemoryInfo. However doing this is only allowed for disjoint formats. One suggestion I have heard was that this should be allowed if the VkImage was created with the tiling format VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT, though this is not specified and doesn't seem to work in practice either. It is unclear anyway why VkBindImagePlaneMemoryInfo duplicates the memory offset info already passed earlier in the VkSubresourceLayout struct. If the imported image is using different offsets than the offsets the VkImage was created with, what would happen?
The de-facto way currently used in wlroots' Vulkan renderer and gamescope is:
This behavior is not well documented and feels not very natural to the API. Hopefully this API can be improved in the future.
xrdesktop's gulkan library contains an example demonstrating this behavior by allocating GPU memory for a format with a drm modifier with 2 memory planes using gbm, exporting each of the planes to an fd and importing those fds into a GulkanTexture.
Interop between Wayland and Vulkan with proper handling of GPU memory is a relatively new topic and the respective Vulkan extensions are only implemented in recent versions of Mesa. Additionally to compile wxrd, a recent version of the Vulkan headers has to be provided by the distribution.
For Building and running wxrd, see the readme.
As a proof of concept we were able to run wxrd on Monado without a 2D window manager running, directly on DRM. By setting the environment variable
XRT_COMPOSITOR_FORCE_VK_DISPLAY=X where X is the number of the display to use you can tell Monado to run on DRM. Support for this and the screen idendifiers can be determined with the vkdisplayinfo tool.
Desktop PCs have become incredibly small. This Ryzen Embedded box can run radv and a fully featured modern Linux graphics stack using a stock x86_64 Arch Linux. It could be hooked up to a battery for a fully portable use case.
Since the Rasberry Pi 4 can run Vulkan through Mesa's v3dv implementation, we gave it a shot to run xrdestkop on it, using Arch Linux ARM for AArch64. Due to an issue where presenting the swapchain to the HMD display fails with the error code VK_ERROR_SURFACE_LOST_KHR we were not able to render to a HMD at the time of writing, though using a regular monitor in direct mode is possible. We also were only able to run Monado with the xrdesktop samples, since wxrd requires the VK_EXT_drm_format_modifier extension, which is currently unavailable for the RPi4.
Thanks to Collabora R&D for funding this work. Get wxrd here: https://gitlab.freedesktop.org/xrdesktop/wxrd
Less than a day away, May 18th looks to be a very busy time. With Live Embedded Event and Embedded Vision Summit taking place almost simultaneously,…
Over the last two years, Bluetooth® audio support has steadily grown in PipeWire and has become a featureful, stable, conformant, open source…
Looking to use hardware-backed and virtual SocketCAN interfaces inside your Kubernetes Pods? A new device plugin now allows processes inside…