Sure, makes perfect sense, I was proposing that before…
So, now I have taken a first full flight with using exclusively your hand tracking layer, with hardware yoke and throttles. There are a couple of issues that became very apparent, that are basically show-stoppers:
-
While hands are tracking great and are always visible, the MSFS “ghost controllers” disappear and appear randomly. Sometimes they just stay for a few seconds and then they’re gone. Seems totally random. I have to clench my fists or “finger gun” several times to make them appear again. Initially I thought that’s the same short timeout MSFS controllers have, but no, it’s much shorter and random.
-
Even worse, when the ghost controller appears again after randomly disappearing, nothing can be highlighted (turned blue or yellow) again. Hands and controllers are tracking, but can’t interact with anything. Switching to laser and back, or using “finger gun” gesture several times, makes it work again, but then after working for a little while, the ghost controller randomly disappears again, and once again the issue is back - it did so every time today. It seems that the code that deals with connecting Leap Motion and your layer works perfectly. It’s the part that feeds it to OpenXR/MSFS as emulated controllers that seem to be quirky. MSFS loses controllers randomly. Maybe you should use some kind of “keep alive” code to prevent ghost controller from disappearing?
Those two issues together makes normal flight practically impossible. They should be fixed, even if other issues that have workarounds, remain. Those issues could only be happening on my system, but I suspect they are universal. It feels like oh so close! You’re almost there at the fully usable software, just need this ironed out. The rest is a bonus.
It’s important to note the positive facts:
-
Hands are tracking great. I can see the “cube hands” all the time my actual hands are in view. They appear instantly when the hand is back in view, and track in a very stable manner. Graphics is smooth and there are no jumps or stutters. It so happens that in my favorite seat position in my motion rig, even my left hand ends up exactly on the yoke, it lined up perfectly with my hardware yoke without trying. That was unexpected and very realistic in a way. Even with rudimentary blocky hand it “connected” me with virtual cockpit more, increasing immersion. So Leap motion controller and your code that feeds on it is very solid.
-
Interaction (when it’s working) is working well, for the most part. Any problems, like rough rotation, seem to be related to MSFS glitches, so once Asobo fixes them - they should work equally well with your layer.
-
The experience of using my hands to operate the cockpit without blindly feeling for a trackball or a controller is AMAZING (during the periods that it works)! It’s a dream come true.
-
“Finger gun” gesture works fairly reliably. The whole experience needs a bit of getting used to, but it looks like after a short while, using index finger as a controller “attachment” and finger gun as a trigger becomes very intuitive. I could press switches and adjust things fairly well. For very fine adjustments I could use my hardware (like GPS dual encoders etc.) But the immersion factor is out of this world!
-
NIS layer on 80% with 40% sharpening (100%OXR / 100% TAA), and disabled “sharpen” in MSFS config file gave me very smooth performance, with noticeably sharper looking than before exterior and interior, while having a very slight increase in shimmering of buildings and straight lines. I think it’s definitely worth it. Overall both graphics and performance are significantly better. If FSR is implemented, it’s interesting to see how it looks. Marrying both layers is a very good idea.
-
GIU works well and is intuitive. Very nice to not have to edit config files and see all changes in flight.
Other, less critical issues and suggestions:
-
When hands go out of view and Leap Motion loses tracking (expected behavior), ghost controller often stays stuck in the last known position (like in front of my face). It should move to a zero position behind/below me instead, or disappear - whatever is easier to code. I shouldn’t remain in view.
Workaround: grab/trigger several times for the ghost controller to appear again and it jumps to wherever the hand is. -
Index finger touching for laser/direct switching doesn’t work, as reported before.
Workaround: assign the wrist touch or other double-handed gesture to that function. It works OK that way. -
Perhaps even darker hand color, like dark gray or even black, should be an option. It may be less distracting. Together with opacity, when it’s implemented, we could dial it to what feels informative enough to see your hands, but without looking too out of place.
-
Translate vibration feedback to a soft “click” sound - for replacing haptic feedback with aural feedback. Using custom sound, and custom sound device would allow to feed the “Buttkicker” and have a true haptic feedback of sorts. I suggested it before, I’m listing some things again just to have it all in a single one place for you and others to discuss.
-
Option to convert a haptic “click” to a trigger press when hand is already in a “finger gun pressed” gesture and approaches a control. That will work as a natural button press.
-
Idea for a new feature, related to 5 above: have 2 modes of aligning a controller. Ghost controller would then jump to desired finger, depending on the position of your hand. For example, mode A: “finger gun pressed” - align the controller it with index finger and press buttons that way as above, or mode B: align it with a thumb or palm while not in a “finger gun” position. That would allow for natural grabs or rotations, and approaching those controls with a “fist clenched, thumb up” position (trigger on thumb down), or something similar. Needs to be tested for convenience, but I think it’s a good idea. It would require 2 sets (2 tabs) of controller alignment settings, and setting a gesture to bind to mode A (default) and mode B. That way we almost have 2 fingers/points active instead of a single point, making controller emulation a step closer to hand emulation.
-
Some may prefer disabling left hand, so there should be a way to switch between laser/direct mode without 2-handed gesture, or even better - still use use 2-handed gesture but just not feed/show the left hand to MSFS. Maybe it already works that way, not sure.
-
Sometimes it’s difficult to keep your hand on the small control, while rotating or moving it. So if possible, maybe some kind of position-locking when trigger is pressed can be implemented, so slight hand movements wouldn’t lose the highlighted point. IT may be a MSFS issue that’s not fixable in your layer, but I’m just listing it because you have surprised me many times already, making things that I thought impossible work.
-
I have to clench my fists or “finger gun” several times to make ghost controllers appear again after they disappeared due to issue above, or a natural timeout. They should appear after a single “grab” gesture or trigger gesture. It takes 3-5 times for them to appear.
I’m very excited by the future of this tool. I will be using it 100% of my flights, even in its current form, but the disappearing controllers / no interaction issue needs to be solved for this to become a reality. Other things can wait or can be bypassed with workarounds, and will be an added bonus.