We're hiring!
*

An xrdesktop summer of code

Lubosz Sarnecki avatar

Lubosz Sarnecki
September 10, 2021

Share this post:

Reading time:

This summer, Christoph Haag and I had the pleasure of taking part in Google Summer of Code (GSoC) as mentors for xrdesktop, the Open Source project bringing the Linux desktop to VR on SteamVR & Monado. Our students, Remco and Manas, were both able to finish their projects and submit merge requests. Kudos!

As our tools of choice, C/GObject and Vulkan were used in both projects, maintaining our goal of keeping xrdesktop low level and thus providing a performant XR experience.

Contributions like these support our vision with projects like xrdesktop and Monado to provide a fully open source XR stack that enables complete control and independence for end users and product builders alike.

Virtual Reality Keyboard

In XR, users want to input text with methods like controllers or hand tracking without having to walk to their desk and use a physical keyboard. The OpenXR API currently does not expose any API for text input, and it is in the applications' responsibility to implement a text input method.

Remco implemented such a virtual keyboard for xrdesktop with support for emojis and 56 languages. Supporting non-english languages and emojis required adding Unicode support to our desktop input synthesis library libinputsynth. The advanced glyph support was achieved by using the Pango font rendering library.


He specified a JSON format for describing keyboard layouts and an importer script that uses the Unicode CLDR Project definitions to achieve internalization. Our current design focus was a simple keyboard with mode switches as seen in the GNOME Shell onscreen keyboard or on mobile phones.

With our internal json format arbitary keyboards can be specified, so one could also replicate a 100% desktop keyboard with numpad or even keyboards that are not feasible in a physical form, like this Japanese keyboard with multiple hundred characters:

Source: https://japan.googleblog.com/2010/04/google.html.


With Remco's work xrdesktop will be able to provide a virtual keyboard internally to xrdesktop, independent of the XR runtime it runs on. We will also be able to expose this keyboard as a stand-alone OpenXR application so it will be usable outside of xrdesktop.

Future work on this project includes implementing input methods for Chinese and other asian languages as well as prediction with suggestions as known from mobile phones and other on-screen keyboards.

More details can be found on Remco's blog post.

GLTF Loading

Manas implemented loading and rendering of GLTF models, which included improving our renderer and adding rendering techniques like normal mapping. This work enables xrdesktop to use assets like VR controller models and 3D environments.

The GLTF loader is based on gthree, the GObject port of three.js.

We are able to parse and render GLTF 2.0 .gtlf and .glb files. With which we will be able to use the controller assets from the W3C Immersive Web WebXR Asset repository in an upcoming release.

Blender's Suzanne model loaded from GLTF.


Future work will make the loader and renderer compatible with the full glTF Sample Model repository from Khronos and provide an asset pipeline from Blender to xrdesktop.

If you are interested in detail's about Manas's project, you can find a summary in his blog post.

Upcoming 0.16 release

These features will get into our next xrdesktop release v0.16, which will be tagged soon. If you are brave enough you can also try our experimental next branches on our freedesktop.org GitLab, following the installation from source instructions from our Wiki.

Don't hesitate to submit issues on our bug tracker or join the discussion on our Discord. Follow @xrdesktop on Twitter.

Comments (5)

    1. Lubosz:
      Sep 14, 2021 at 03:28 PM

      Layouts like this can be defined without problem in our JSON format:
      https://gitlab.freedesktop.org/xrdesktop/xrdesktop/-/blob/next/res/keyboard-layouts/small.json

      Currently we are only providing the common layouts from the Unicode CLDR Project, but it should be easy to add new ones.

      Reply to this comment

      Reply to this comment

    1. Lubosz:
      Sep 14, 2021 at 03:28 PM

      Thanks for finding a better source! We'll update the post.

      Reply to this comment

      Reply to this comment


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

The latest on cmtp-responder, a permissively-licensed MTP responder implementation

12/06/2024

Part 3 of the cmtp-responder series with a focus on USB gadgets explores several new elements including a unified build environment with…

A roadmap for VirtIO Video on ChromeOS: part 3

06/06/2024

The final installment of a series explaining how Collabora is helping shape the video virtualization story for Chromebooks with a focus…

Hacking on the PipeWire GStreamer elements

05/06/2024

Last week I attended the GStreamer spring hackfest in Thessaloniki to work on the PipeWire GStreamer elements and connect with the community.

Transforming speech technology with WhisperLive

28/05/2024

The world of AI has made leaps and bounds from what It once was, but there are still some adjustments required for the optimal outcome.…

Re-converging control flow on NVIDIA GPUs - What went wrong, and how we fixed it

25/04/2024

While I managed to land support for two extensions, implementing control flow re-convergence in NVK did not go as planned. This is the story…

Automatic regression handling and reporting for the Linux Kernel

14/03/2024

In continuation with our series about Kernel Integration we'll go into more detail about how regression detection, processing, and tracking…

Open Since 2005 logo

Our website only uses a strictly necessary session cookie provided by our CMS system. To find out more please follow this link.

Collabora Ltd © 2005-2024. All rights reserved. Privacy Notice. Sitemap.