After a two-year hiatus, FOSS XR took place in Minneapolis this past October. Besides being a wonderful place to come together and meet many different awesome people working on open-source XR, the conference held several talks directly related to Monado, our open-source OpenXR runtime. In this post, I'll focus on summarizing some of the key points of the "Visual-inertial tracking for Monado" talk as an overview of its current state.

In my last blog post, I took a more in-depth look at the details of integrating visual-inertial tracking solutions with Monado and why it was an important component to build. The gist is that new headsets have been coming with only cameras and IMUs as their sensors for tracking, and there hasn't been a clear alternative to their proprietary tracking solutions. Although still not meant for final users, thanks to this module (and thanks to Basalt in particular), we can now get OpenXR apps tracked on a totally open-source software stack on Linux.

The open drivers in Monado that leverage this tracking component are:

• wmr: tested on the Odyssey+ HMD, but the community has reported other headsets like the Reverb G1 working as well. New ones should mostly work out of the box except for some minor tuning.
• rift_s: thanks to the awesome work from Jan Schmidt (@thaytan) building reverse-engineered open drivers, the Rift S is now supported in Monado, and takes advantage of our visual-inertial tracking too.
• vive: tested on the Index. Although still in the early stages, it's quite cool to be able to track the headset without using external lighthouses!
• depthai: tested with the NorthStar HMD. This is the first AR headset using this component and I had the pleasure of seeing it working for the first time at FOSS XR together with the rift_s :)
• realsense: tested with the RealSense D455. These are computer vision cameras similar to depthai. They can be used as an attachment to other devices for tracking.

Below are some videos of me using the devices with Monado:

• Samsung Odyssey+ running Monado with Basalt and HT
  
  

• Valve Index running Monado with Basalt
  
  

• Intel RealSense D455 running Monado with Basalt
  
  

• Visual-inertial tracking for Monado demo at FOSS XR
  
  

Some adjacent, but also important developments are the metrics pipeline we have for measuring different systems or to measure improvements we do directly on top of Basalt. Given that the research area is quite active in developing new techniques and implementations, the VI module is designed to be flexible enough to integrate and measure these new breaktrhough systems. Meanwhile, we have plans to continue improving Basalt, to create XR-specific metrics, and to make (and publish!) XR-specific visual-inertial datasets with lighthouse ground-truth. In the long run, we also plan to handle dynamic objects, to integrate them into the optimization, to jointly optimize with our other trackers like our hand-tracking, to do online calibration, and even to start semantically understanding the device's environment.

It's still worth noticing that there is more work to do to get closer to commercial solutions, but what we have right now works well enough for many use cases, and it's completely open source. Furthermore, we have a good infrastructure in place for additional development, and it's mostly a matter of iterations until we reach even better results.