Mateo de Mayo
December 20, 2022
After a two-year hiatus, FOSS XR took place in Minneapolis this past October. Besides being a wonderful place to come together and meet many different awesome people working on open-source XR, the conference held several talks directly related to Monado, our open-source OpenXR runtime. In this post, I'll focus on summarizing some of the key points of the "Visual-inertial tracking for Monado" talk as an overview of its current state.
In my last blog post, I took a more in-depth look at the details of integrating visual-inertial tracking solutions with Monado and why it was an important component to build. The gist is that new headsets have been coming with only cameras and IMUs as their sensors for tracking, and there hasn't been a clear alternative to their proprietary tracking solutions. Although still not meant for final users, thanks to this module (and thanks to Basalt in particular), we can now get OpenXR apps tracked on a totally open-source software stack on Linux.
The open drivers in Monado that leverage this tracking component are:
wmr: tested on the Odyssey+ HMD, but the community has reported other headsets like the Reverb G1 working as well. New ones should mostly work out of the box except for some minor tuning.
rift_s: thanks to the awesome work from Jan Schmidt (@thaytan) building reverse-engineered open drivers, the Rift S is now supported in Monado, and takes advantage of our visual-inertial tracking too.
vive: tested on the Index. Although still in the early stages, it's quite cool to be able to track the headset without using external lighthouses!
depthai: tested with the NorthStar HMD. This is the first AR headset using this component and I had the pleasure of seeing it working for the first time at FOSS XR together with the
realsense: tested with the RealSense D455. These are computer vision cameras similar to
depthai. They can be used as an attachment to other devices for tracking.
Below are some videos of me using the devices with Monado:
Some adjacent, but also important developments are the metrics pipeline we have for measuring different systems or to measure improvements we do directly on top of Basalt. Given that the research area is quite active in developing new techniques and implementations, the VI module is designed to be flexible enough to integrate and measure these new breaktrhough systems. Meanwhile, we have plans to continue improving Basalt, to create XR-specific metrics, and to make (and publish!) XR-specific visual-inertial datasets with lighthouse ground-truth. In the long run, we also plan to handle dynamic objects, to integrate them into the optimization, to jointly optimize with our other trackers like our hand-tracking, to do online calibration, and even to start semantically understanding the device's environment.
It's still worth noticing that there is more work to do to get closer to commercial solutions, but what we have right now works well enough for many use cases, and it's completely open source. Furthermore, we have a good infrastructure in place for additional development, and it's mostly a matter of iterations until we reach even better results.
We can now confidently say that PipeWire is here to stay. But of course it is not the end of the journey. There are many new areas to explore…
Our look at the Rust crate for interconnected objects continues, as we examine how persian-rug really does tie the room together by providing…
The testing ecosystem in the Linux kernel has been steadily growing, but are efforts sufficiently coordinated? How can we help developers…
With the upcoming 0.5 release, WirePlumber's Lua scripts will be transformed with the new Event Dispatcher. More modular and extensible…
This second installment explores the Rust libraries Collabora developed to decode video and how these libraries are used within ARCVM to…
Why is creating object graphs hard in Rust? In part 1, we looked at a basic pattern, where two types of objects refer to one another. In…