Hand Tracking in MSFS - Leap Motion / UltraLeap native support for WMR/OpenXR

AuthentiKit is everything, but starting with Spitfire. The idea is to make a platform for easy replica interchangeable flight controls. Kind of how WordPress accelerated website development.

Very handy - the fact that leap motion has blind spots exactly where most h/w flight controls would be, because, as you say I mainly want hand tracking for the instrument panel and all those small things like VOR that aren’t worth building controls for unless you’re really dedicated to a particular aircraft.

Having said I’m not interested in instrument panels - there is a half way house where you can model certain regularly used controls physically and use hand tracking for the rest. That’s the thinking behind this early AuthentiKit build of a DR400 setup.

1 Like

Very interesting! I use the opposite approach, that shares the “most used controls in hardware + the rest with hand tracking” concept with your approach. I have built a Frankenstein of a motion cockpit that is convertible for most type of sims and aircraft, but has the hardware that’s not mapped precisely to the VR cockpit locations, as that is virtually impossible to do for many different types of aircraft. Instead I rely on muscle memory locating anything I need, even if it doesn’t map to the visuals. Layouts are logically based, and mapped close to what they would be on a real aircraft, even if they are not located at the same precise spot.

Yoke and joystick conversion does map very well, actually - I was shocked that my “hand” with the @mbucchia tool hugged the yoke precisely. You can see the 3D printed joystick configuration on the photo. The rig is a work in progress but it’s basically fully functional now. The only factory made parts are the pedals and a cannibalized yoke grip. You can see the yoke on the floor, it converts in 1 minute. Both yoke and joystick have a large throws and are very accurate. I used hall sensors on a large pendular yoke (full 180 degree rotation), with a joystick conversion kit as a secondary, but it works remarkably well because of range of motion and precise feel. It doesn’t feel like a computer joystick or a yoke at all. Tension is done with bungee cords - very smooth, without any central detent but a distinct gradual tension when moved.

You can see throttle quadrant I built that’s mostly Boeing 737 inspired (Dual 737 throttles, including A/T disconnect and TO/GA buttons, analog reverser levers, speedbrake and flaps levers), encoder-based trim wheel, but with added Mix and Prop levers (that also can be spit in to right and left control with a modifier button press), and several buttons and a rotational switch to use for magnetos etc. That’s used in GA and airliners. Beside it is the Wacom tablet to write down ATC and look at the charts etc.

On the left side is a HOTAS-type setup used in military aircraft that have joystick control and need throttle on the left. It has analog thumbstick added on the right (useful in space sims like Elite Dangerous for thruster control). With a simple pin replacement Throttle can be pushed down and becomes a Collective for helicopter use! It has a pot under the thumb to be used for the RPM control. And there’s a switch box next to it for left-hand use.

The center has a switchbox on the right, for all kinds of lights, 3-position lockable Gear lever (pull to move), 2 analog pots etc. Works for all aircraft. Next to it is a 6 dual encoder + 8-button and a 3-pos switch box that controls all of the autopilot rotators and analog instrument knobs or GPS unit knobs + main GPS and AP buttons. I cam tune radio and transponder freqs etc. with ease. Very easy to dial G1000 or G3000 units or airliner autopilots. Close together so no problem feeling what encoder I’m using, they are also different shapes, just like G1000 knobs. And a keyboard + trackball for when they are unavoidable.

So in any aircraft I know where can I control all the instruments I need - they will always be in the same place. What’s not mapped to hardware - that’s where hand tracking is perfect - pressing buttons and lots of other interactions with GPS/FMS etc. So those two approaches together would give the total freedom and control in flight. It’s impossible to map everything to hardware, and very challenging to build a 1:1 map - at the same time limiting to a single aircraft layout, or in your case, a few, with a non-trivial conversion. My rig is at the limit of what can be easily remembered - both by muscle memory and just simply remembering what is mapped where. Hand tracking will save the day.

This whole arrangement is on the 2DOF moving frame that gives motion cues based on roll/pitch accelerations and roll/pitch position, including mixed in feedback for surge (acceleration), sway, yaw, “road noise” etc.

2 Likes

I recognise those windscreen wiper motors. I didn’t self build my 3DOF - went with DOF Reality when they were just starting out, but after spending time in x-simulator and getting my head around it all I added a nice g-seat of my own design. Another project for you :slight_smile:

1 Like

Hi - The Aero uses it’s own OpenXR I have discovered. Does that create a problem integrating your s/w or will it inject the image independently of that?

I can only test with the devices I have. So far we’ve been lucky that my software has been working with all the headsets it’s been tested with. However, I’ve had reports that my NIS software does not work with the Aero. I’m investigating this issue now, but it might take longer before I can make it work.

3 Likes

Hi gents, this is incredible work. I’ve flagged this up to our XR team to show them what you guys have been working on using just the tools that are out there!

Is it possible to get the OP updated with a tl;dr instructions/how-to on the current builds you guys have? This thread has turned into quite a beast!

Just on the subject of hand tracking implementation theory, one thing that we’ve found works really well in flight sim interactions, but which is unfortunately out of the scope of what is possible here, is adding physics to the hands so instead of passing through cockpit elements like ghosts, they interact with them like they would in real life. It’s not as cheap computationally and needs work from the developer (and engine support), but it is definitely the ideal setup in our experience and helps a lot with accuracy and overshoot on controls (especially in bumpy flights!).

3 Likes

Thanks! I was going to bring this up to your team eventually. After all we are in the same meeting every Thursday (the Khronos working group meeting) :slight_smile:

I’m just a little embarrassed by the v0 of this tool, and wanted to wait for the next version, which is a clean rewrite, all nice code, much better for other devs to look at :slight_smile:

I’ll be happy to. Let @mbucchia point me to the best text or post in this thread that would serve as the TLDR summary and I will update the top post. I did update it when this tool was released but not with a detailed instructions.

This would be the best, but I bet MSFS is not even close in terms of interaction. Not much of a chance of physics modeling in switches to be “bumped into”. I think the approximation of this effect is what they did in DCS, but it’s clunky and unpredictable. You bump your hand against a switch, and never know which way it would go and how many times it fires :-).

But I did propose a workaround that should work well to mimic this effect: the vibration “tick” feedback when your hand enters the switch’s control/interaction area could be detected by @mbucchia layer and a “click” fired once. That would mean in practice that you bump your finger against the button and it would press, or switch would trigger (only once, not all over the place like in DCS). For multi-position switches each “finger bump” would click it once. It would be very manageable. And as usual, things could still be activated with a “grab” or “trigger finger” gestures.

Right now, to trigger a switch or press a button, you need to bring your finger there (assuming the controller point is tied to your index finger, which is how I set it up), then while holding your hand in this very small control area you need to bend index finger to generate a “click”, without losing the area from focus, which can be tricky. But if my suggestion is implemented, you’d just flick the switch in one easy motion, and there won’t be false repeated triggers until you are away and back again. This probably should be an option that’s on by default, but that could be disabled in settings. As a bonus, on the detected vibration “tick” the layer can fire a soft “click” sound, which would serve as a audible and/or haptic feedback (if routed to ButtKicker transducer, which many use - I have a DIY version of it), replacing the controller “tick” which is obviously missing otherwise if controllers are replaced by hands. Yes, you’d feel your finger touching a switch in your butt, which is admittedly weird, but it beats not having any haptics at all :slight_smile:

Those are PGSaw motors from ebay - much more powerful than most (if not all) wiper motors, with a 50:1 gear ratio. They are used for sawmills I think. They are 12V 180 Watt (0.24HP) and pull 60NM Load. G-seat is nice, maybe I’ll get to building it some day, but it feels so nice even now, that I’m good for now.

1 Like

I’m at the stage where everything else looks fixed, and this is the one issue I have not looked into yet. I probably don’t spend enough time in the Sim to hit this issue…

“Seems totally random” that’s making it really hard to look into. Are you having the issue in the middle of using the hands or is it after hands come back into view after being inactive?

Are you able to reproduce the issue with the VR controllers and not the hands?

This started to happen right away. By “random” I mean that in 15-20 minutes I had it happen like 10 times, just at unpredictable moments. It may be happening each time Leap Motion lost tracking, but I think it just happened in front of eyes, so that should be there with solid tracking.

I didn’t try, but I think I will fire it up now and do some testing with it. I know that DCS tracking was solid, so Leap Motion is functioning well. I will report back later tonight. I will try just the controllers, and then with the layer and Leap Motion.

Tested with VR controllers - totally stable. No issues. Will proceed with testing with Leap Motion.

1 Like

Tested it with hand tracking - same issue, extremely bad. While VR controllers don’t have any problems, what happens with hand tracking is the “ghost controllers” often disappear, sometimes when hand is still perfectly visible and tracking, just the controller image is gone, along with any interaction. Or they can disappear together when hand is out of the tracking zone or obstructed. Just try going through the engine start checklist and start a TBM for example. while doing it I had this issue happen 10 or 20 times in 15 minutes.

  • Either just the controller disappears randomly and often, or it disappears together with the hand
  • To get it back I have to clench my fist several times, but often when I get it back it doesn’t activate anything anymore
  • To get past that I have to trigger laser then back to direct mode. Then it starts working, but within a minute or two it will always disappear again. And when it appears - if more than a few seconds have passed, it’s almost certainly will not interact again.

If you can reproduce this issue at all (I think other people did), you will see it within 5 minutes for sure. I could not even get to take-off that way. It’s just too frustrating to do this whole sequence every 2-3 steps of a checklist or when anything else needs to be operated. Almost every time, when I had to move my hand out of the frame to touch a throttle, it won’t interact when I get it back.

I tried to establish some definite steps to make it stay or disappear, so you could verify that. But it’s so chaotic and it kept glitching so much that I couldn’t figure out what triggers it. I tried even to keem my hand always in front for a while, to make sure it still happens when tracking is not lost - and it did.

Hand images are tracking stable, when hand is in the frame. Often hands are not in the frame - on my lap or throttle or yoke, so they are not tracking. But when I bring my hands forward, they track stable, so it’s not a Leap Motion problem.

I think maybe some “keep alive” will solve that? When hand is not tracking, even for a second, controller should just jump to the floor behind me, or something like that, instead of disappearing, but be kept alive.

1 Like

So it sounds like it happens prominently when the hands are not visible to the camera?

I’ll try again myself tomorrow and add some stats on screen to dump the state.

Maybe there are occasional frames reporting loss of tracking and that somehow messes the game up. Because if you are seeing the skeleton hands, it’s literally the same exact data that is presented to the game for the input. But it looks like the game does not see the input at all, or I think not accepting the input.

I can try to add a low pass filter (aka debouncing) on the valid/tracked flags. It won’t solve the issue of the dangling ghost controller when hands are not tracked.

For that a keep alive with a fake position is doable, though not a generic solution. Maybe if we remove the model file for the ghost controller it won’t be an issue.

Thanks for all the input as always!

1 Like

Actually no, it happens all the time. Even if I try to keep my hands visible and tracking all the time, I can’t prevent it from happening, and it still happens often. I just happens more certainly if hands are lost.

One important detail: when the controller disappears and I clench my fist so it appears again, often after it appears it can’t interact (no highlighting or interaction happens), but the “ghost controller” model is visible, including the white arrow. So it LOOKS as if it should work, but it still doesn’t. I always use them in derect mode, not laser. So, to clarify:

  1. Controllers randomly and often disappear (controller model invisible, hand model is tracking and visible)
  2. When they appear again (after several trigger/grab gestures), they often can’t interact with anything (model still visible!), until tapped to laser mode and back to direct mode again. Then they start interacting.

I have the same issue here, also seems to be stochastic.
I first taught it was more often seen on some aircraft, but not sure at all.

I’m afraid it’s perhaps the same issue (MSFS) I have with mouse interaction, when the mouse lag, stop move and do not follow correctly real mouse input, only inside the VR world.

But I’m happy to know that you don’t see it with real controller hardware (which I don’t have). It means there is a good chance as you mentioned, to find a workaround, maybe maintaining a fake controller position at its last seen position before leaving the camera field.

In the meantime and for this issue, I stopped trying to use it for other goals than experimentation sadly.

Yes, all the glitchiness is the layer only, Reverb G2 controllers are working rock-solid. MSFS does have an issue with them disappearing after a while, but I stress that it’s not the same issue as with handtracking controllers disappearing: it happens always after the same time (not randomly), which is much longer, and reactivation needs a single squeeze (not repeated), and interaction works perfectly every time after reactivation. So that is not a bug, just an unfortunate decision on the Asobo’s side to hide the controllers too soon after a period of inactivity. But they are behaving consistently - disappear after a certain timeout and work perfectly when they reappear.

Yes, in current state it’s unusable because of that issue. I have to literally use the sequence of several hand squeezes (to wake up disappeared controller), then several wrist taps to get laser and back to direct mode to be able to interact, only to loose it all within a minute anyway and do it all over again. But this is early beta, so this is expected behavior. The proof of concept is fantastic! I’m sure @mbucchia can iron out those bugs, they seem trivial by comparison, and merged in the single tool this would be awesome.

If later he can incorporate triggering controls based on haptic “ping” when hand is in “finger gun” position, this would be even better than DCS with its native Leap Motion integration, in my experience at least. You can press buttons in DCS by bumping into them, but in any state, so it happens unpredictably, and there is no “denoise” so you can trigger single switch 5 times in 1 second accidentally because of the hand jitter. What needs to happen is “bumping” only in “finger gun” pose (settable in config), and only a single click after a single haptic “ping”. That way you only trigger once per touch, and have to move out of the interaction zone and back to click again. This would be so great, I can’t wait.

I have the exact same behaviour here, good to know I’m not alone :wink:
POC is brilliantly done and you are right, it’s the path for the future.

It’s finally a real VR experience! Unlike many people who came from pancake sims and static cockpits with TrackIR at best, that can be happy with a mouse in VR, I am a VR enthusiast (yes, in addition to using MSFS since about '95 on and off) and a VR just being a camera without a true VR was a tragedy for me, that why I called for VR controller implementation on day one (it was my post that got 500+ votes and #1 place in VR developer snapshot). VR controllers have really improved immersion, but thy still are a crutch for using your hands, just a much smaller crutch than a mouse. That’s why I called for hand tracking support - it’s a logical next step. When it became real - thanks to @mbucchia - even right now that it works for brief time, I can tell that this is what a real VR is. It’s the most convenient and immersive way to experience VR. Tracking will not be as reliable as controllers of course, but it’s worth it in huge immersion increase, and just getting rid of a huge immersion killer of searching for the mouse or the controller before any interaction.

3 Likes

Will this work with SteamVR? I would like to use it with VTOL VR and their new helicopter. Sorry if this has already been answered, I looked in the thread and didn’t find an answer.