We're hiring!

State of Monado's visual-inertial tracking

Mateo de Mayo avatar

Mateo de Mayo
December 20, 2022

Share this post:

Reading time:

After a two-year hiatus, FOSS XR took place in Minneapolis this past October. Besides being a wonderful place to come together and meet many different awesome people working on open-source XR, the conference held several talks directly related to Monado, our open-source OpenXR runtime. In this post, I'll focus on summarizing some of the key points of the "Visual-inertial tracking for Monado" talk as an overview of its current state.

In my last blog post, I took a more in-depth look at the details of integrating visual-inertial tracking solutions with Monado and why it was an important component to build. The gist is that new headsets have been coming with only cameras and IMUs as their sensors for tracking, and there hasn't been a clear alternative to their proprietary tracking solutions. Although still not meant for final users, thanks to this module (and thanks to Basalt in particular), we can now get OpenXR apps tracked on a totally open-source software stack on Linux.

The open drivers in Monado that leverage this tracking component are:

  • wmr: tested on the Odyssey+ HMD, but the community has reported other headsets like the Reverb G1 working as well. New ones should mostly work out of the box except for some minor tuning.
  • rift_s: thanks to the awesome work from Jan Schmidt (@thaytan) building reverse-engineered open drivers, the Rift S is now supported in Monado, and takes advantage of our visual-inertial tracking too.
  • vive: tested on the Index. Although still in the early stages, it's quite cool to be able to track the headset without using external lighthouses!
  • depthai: tested with the NorthStar HMD. This is the first AR headset using this component and I had the pleasure of seeing it working for the first time at FOSS XR together with the rift_s :)
  • realsense: tested with the RealSense D455. These are computer vision cameras similar to depthai. They can be used as an attachment to other devices for tracking.

Below are some videos of me using the devices with Monado:

  • Samsung Odyssey+ running Monado with Basalt and HT

  • Valve Index running Monado with Basalt

  • Intel RealSense D455 running Monado with Basalt

  • Visual-inertial tracking for Monado demo at FOSS XR

Some adjacent, but also important developments are the metrics pipeline we have for measuring different systems or to measure improvements we do directly on top of Basalt. Given that the research area is quite active in developing new techniques and implementations, the VI module is designed to be flexible enough to integrate and measure these new breaktrhough systems. Meanwhile, we have plans to continue improving Basalt, to create XR-specific metrics, and to make (and publish!) XR-specific visual-inertial datasets with lighthouse ground-truth. In the long run, we also plan to handle dynamic objects, to integrate them into the optimization, to jointly optimize with our other trackers like our hand-tracking, to do online calibration, and even to start semantically understanding the device's environment.

It's still worth noticing that there is more work to do to get closer to commercial solutions, but what we have right now works well enough for many use cases, and it's completely open source. Furthermore, we have a good infrastructure in place for additional development, and it's mostly a matter of iterations until we reach even better results.

Comments (0)

Add a Comment

Allowed tags: <b><i><br>Add a new comment:

Search the newsroom

Latest Blog Posts

Thoughts on PipeWire 1.0 and beyond


We can now confidently say that PipeWire is here to stay. But of course it is not the end of the journey. There are many new areas to explore…

Persian Rug, Part 3 - The warp and the weft


Our look at the Rust crate for interconnected objects continues, as we examine how persian-rug really does tie the room together by providing…

Advocating a better Kernel Integration for all


The testing ecosystem in the Linux kernel has been steadily growing, but are efforts sufficiently coordinated? How can we help developers…

WirePlumber: Exploring Lua scripts with Event Dispatcher


With the upcoming 0.5 release, WirePlumber's Lua scripts will be transformed with the new Event Dispatcher. More modular and extensible…

A roadmap for VirtIO Video on ChromeOS: part 2


This second installment explores the Rust libraries Collabora developed to decode video and how these libraries are used within ARCVM to…

Persian Rug, Part 2 - Other ways to make object soups in Rust


Why is creating object graphs hard in Rust? In part 1, we looked at a basic pattern, where two types of objects refer to one another. In…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2023. All rights reserved. Privacy Notice. Sitemap.