We're hiring!

Monado's "Mercury" hand tracking now ready for use!

Moses Turner avatar

Moses Turner
February 24, 2023

Share this post:

Reading time:

In our previous blog post, I teased our new optical hand tracking pipeline, codenamed Mercury. Work on this tracking method began around January 2022.

Now, after a little over a year of development, it's finally ready for the public to use!

Mercury currently works on Valve Index, Windows Mixed Reality headsets (Reverb G2 v2 shown below), Oculus Rift S (early support), and Luxonis cameras (video shows a OAK-D Pro W Dev mounted on a North Star). It estimates hand pose accurately, supports limited hand-over-hand interactions, tracks fast hand movements, and is useable for drawing, typing and UI interaction.

Usage documentation is here if you want to run it in Monado. Expect in the coming few weeks some improvements to hand tracked-state stability, reductions to background jitter, a SteamVR driver so you can use it on Windows, and a much more detailed blog post showcasing the work we did to get here!


Comments (4)

  1. Roman:
    Mar 01, 2023 at 05:33 AM

    This is very exciting! I hope SteamVR driver is coming that would let it be used in the similar way as UltraLeap does it, only it has so much better potential, as no extra hardware is needed, and HMDs cameras are wide-angle and perfectly aligned. Whe SteamVR driver is ready, would it work with existing WMR OpenXR runtime, or would it replace it? Or would it only work with StemVR OpenXR runtime (which is inferior for Reverb G2 and other native WMR HMDs)?

    Reply to this comment

    Reply to this comment

  2. RalfMarco:
    Mar 01, 2023 at 10:17 AM

    This is fantastic!

    A openXR toolkit driver would be amazing. For MSFS Community and VR Pilots. This Handtracking in an Airplane - Game changer! Many fly with the reverb G2.

    Reply to this comment

    Reply to this comment

  3. hyperdrive:
    Mar 04, 2023 at 11:03 PM

    will it work while you are holding ANY type of controllers, so you can have access to thumb sticks and buttons while still having full finger tracking? otherwise, it's kind of useless because I shouldn't have to give up certain controller necessities for a small but wanted/fun gimic that most games don't even support. I say this because the quest 2 has this issue, and time and time again I see people on the quest 2 try to do simple things like walking and jumping while using the full finger tracking and I see that as a big problem and should have never happened in the first place.

    Reply to this comment

    Reply to this comment

Add a Comment

Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest News & Events

Monado accepted for XROS 2023!


We're proud to announce that Monado, the free and open source XR platform, has been accepted as a mentoring organization for XROS, the XR…

Showcasing the STM32MP1 at Embedded World


As a recent new member of STMicroelectronics' Partner Program, we're excited to be showcasing the STM32MP1 at Embedded World this week,…

Connecting at Embedded World 2023


Nestled in the historic city of Nuremberg, the annual Embedded World conference will be taking place from March 14 to 16. Collabora will…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2023. All rights reserved. Privacy Notice. Sitemap.