Hand Tracking in MSFS - Leap Motion / UltraLeap native support for WMR/OpenXR

Here’s a video showing my latest progress (sorry the quality/lighting isn’t the best):

Progress update on MSFS 2020 with Windows Mixed Reality and Leap Motion hand tracking - YouTube

You can see the hands being displayed in the cockpit, using “left index to right index tap” to emulate the B button (switching between ray mode and direct interaction), “left wrist tap” to go to the menu (but could be replaced with Windows button too in the config).

Then I go a grab the altimeter calibration knob by “pinching” then turning, which feels pretty natural, also use the dashboard dimmer in the same way. It’s a little bit hard to see in the video, but you can see the altimeter needle move and the lights dimming. I use the same pinching to grab and pull the throttle.

You can then see me use “squeeze” to switch hand focus (switch which hand is active), and release the brakes (which did take me a couple of tries, I think the squeeze detection needs a bit of tweaking). I obviously have no clue what I’m doing in the cockpit so I start swerving to the side of the runway, then I visibly panic and pause to the menu :smiley: :smiley: :smiley:

Here are also some screenshots of the configuration tool showing what can be tweaked:




I’ve implemented a “thumb press” action which is what @RomanDesign you suggested in your latest post, but haven’t tried with it yet. Definitely something to experiment with and compare with the pinching!

Fairly happy with the handling so far! I did find a quite noticeable bug that will delay the beta release: in the headset, there is an error in projecting the hands so the way they are drawn is a bit blurry. But I think I’ll be able to fix this quickly. I’m hoping to release the beta tomorrow or Saturday.

2 Likes