This summer, Christoph Haag and I had the pleasure of taking part in Google Summer of Code (GSoC) as mentors for xrdesktop, the Open Source project bringing the Linux desktop to VR on SteamVR & Monado. Our students, Remco and Manas, were both able to finish their projects and submit merge requests. Kudos!

As our tools of choice, C/GObject and Vulkan were used in both projects, maintaining our goal of keeping xrdesktop low level and thus providing a performant XR experience.

Contributions like these support our vision with projects like xrdesktop and Monado to provide a fully open source XR stack that enables complete control and independence for end users and product builders alike.

Virtual Reality Keyboard

In XR, users want to input text with methods like controllers or hand tracking without having to walk to their desk and use a physical keyboard. The OpenXR API currently does not expose any API for text input, and it is in the applications' responsibility to implement a text input method.

Remco implemented such a virtual keyboard for xrdesktop with support for emojis and 56 languages. Supporting non-english languages and emojis required adding Unicode support to our desktop input synthesis library libinputsynth. The advanced glyph support was achieved by using the Pango font rendering library.

He specified a JSON format for describing keyboard layouts and an importer script that uses the Unicode CLDR Project definitions to achieve internalization. Our current design focus was a simple keyboard with mode switches as seen in the GNOME Shell onscreen keyboard or on mobile phones.

With our internal json format arbitary keyboards can be specified, so one could also replicate a 100% desktop keyboard with numpad or even keyboards that are not feasible in a physical form, like this Japanese keyboard with multiple hundred characters:

Source: https://japan.googleblog.com/2010/04/google.html.

With Remco's work xrdesktop will be able to provide a virtual keyboard internally to xrdesktop, independent of the XR runtime it runs on. We will also be able to expose this keyboard as a stand-alone OpenXR application so it will be usable outside of xrdesktop.

Future work on this project includes implementing input methods for Chinese and other asian languages as well as prediction with suggestions as known from mobile phones and other on-screen keyboards.

More details can be found on Remco's blog post.

GLTF Loading

Manas implemented loading and rendering of GLTF models, which included improving our renderer and adding rendering techniques like normal mapping. This work enables xrdesktop to use assets like VR controller models and 3D environments.

The GLTF loader is based on gthree, the GObject port of three.js.

We are able to parse and render GLTF 2.0 .gtlf and .glb files. With which we will be able to use the controller assets from the W3C Immersive Web WebXR Asset repository in an upcoming release.

Blender's Suzanne model loaded from GLTF.

Future work will make the loader and renderer compatible with the full glTF Sample Model repository from Khronos and provide an asset pipeline from Blender to xrdesktop.

If you are interested in detail's about Manas's project, you can find a summary in his blog post.

Upcoming 0.16 release

These features will get into our next xrdesktop release v0.16, which will be tagged soon. If you are brave enough you can also try our experimental next branches on our freedesktop.org GitLab, following the installation from source instructions from our Wiki.

Don't hesitate to submit issues on our bug tracker or join the discussion on our Discord. Follow @xrdesktop on Twitter.