We're hiring!

Monado's "Mercury" hand tracking now ready for use!

Moses Turner avatar

Moses Turner
February 24, 2023

Share this post:

Reading time:

In our previous blog post, I teased our new optical hand tracking pipeline, codenamed Mercury. Work on this tracking method began around January 2022.

Now, after a little over a year of development, it's finally ready for the public to use!

Mercury currently works on Valve Index, Windows Mixed Reality headsets (Reverb G2 v2 shown below), Oculus Rift S (early support), and Luxonis cameras (video shows a OAK-D Pro W Dev mounted on a North Star). It estimates hand pose accurately, supports limited hand-over-hand interactions, tracks fast hand movements, and is useable for drawing, typing and UI interaction.

Usage documentation is here if you want to run it in Monado. Expect in the coming few weeks some improvements to hand tracked-state stability, reductions to background jitter, a SteamVR driver so you can use it on Windows, and a much more detailed blog post showcasing the work we did to get here!


Comments (7)

  1. Roman:
    Mar 01, 2023 at 05:33 AM

    This is very exciting! I hope SteamVR driver is coming that would let it be used in the similar way as UltraLeap does it, only it has so much better potential, as no extra hardware is needed, and HMDs cameras are wide-angle and perfectly aligned. Whe SteamVR driver is ready, would it work with existing WMR OpenXR runtime, or would it replace it? Or would it only work with StemVR OpenXR runtime (which is inferior for Reverb G2 and other native WMR HMDs)?

    Reply to this comment

    Reply to this comment

  2. RalfMarco:
    Mar 01, 2023 at 10:17 AM

    This is fantastic!

    A openXR toolkit driver would be amazing. For MSFS Community and VR Pilots. This Handtracking in an Airplane - Game changer! Many fly with the reverb G2.

    Reply to this comment

    Reply to this comment

  3. hyperdrive:
    Mar 04, 2023 at 11:03 PM

    will it work while you are holding ANY type of controllers, so you can have access to thumb sticks and buttons while still having full finger tracking? otherwise, it's kind of useless because I shouldn't have to give up certain controller necessities for a small but wanted/fun gimic that most games don't even support. I say this because the quest 2 has this issue, and time and time again I see people on the quest 2 try to do simple things like walking and jumping while using the full finger tracking and I see that as a big problem and should have never happened in the first place.

    Reply to this comment

    Reply to this comment

  4. Igor2402:
    May 25, 2023 at 05:10 PM

    Can you make a tutorial...
    How can i use this in Rift S, unfortunately i am layman in this things...
    But i really wanna use this feature. Please, make a Tutorial Video for windows 10

    Reply to this comment

    Reply to this comment

  5. Matt Geders:
    Sep 07, 2023 at 09:05 PM

    Any update on the development for Windows 10/SteamVR?

    Reply to this comment

    Reply to this comment

    1. Jakob Bornecrantz:
      Sep 08, 2023 at 02:45 PM

      You might be able to compile it yourself, changes will probably needed. But there are no ready made installable solution, we have no immediate plans to make one. Hopefully somebody in the community can step up and do that.

      Reply to this comment

      Reply to this comment

Add a Comment

Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest News & Events

FOSDEM 2024 - Recorded presentations (videos) now available


Collabora's engineers presented six talks over the course of the weekend, with topics including a review of recent improvements to GStreamer,…

A framework to share analytics data in GStreamer


Engineers have widely adopted GStreamer to build video analytics pipelines, and while many companies have indeed built their machine learning…

Wine on Wayland: A year in review (and a look ahead)


2023 was a great year for the Wayland driver for Wine. After several merge requests, many people are now already able to use the latest…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2024. All rights reserved. Privacy Notice. Sitemap.