We hope they implement the controllers in vr, they are essential but also the use of multiple windows at the same time as it was in fsx, as well as better flight physics, to date the only sim a bit faithful to reality is xplain, take an example
I’ve posted this a few days ago:
I’ve found traces of voice recognition bits in the binaries… Maybe just experimental implementation left over, maybe actual future version bits, I don’t know.
Same for VR: there is no controller support yet, but there are already bits telling me they are experimenting with controlling the switches and knobs via a laser pointer or by direct contact. The latest update also includes the 3D model for hands:
1 Like
I hoped even before release that voice recognition would be used. Years old Pilot2ATC software uses it wonderfully for real communication with AI ATC contollers and that works great (for the most part) - such a liberating experience! Also, I used free VoiceMacro software to also assign all kind of commands via key presses and TTS. So I could say “Gear Down” and gear would come down, ask FO to read specific checklist etc. It’s all demonstrated in my video on top post. And it’s all done with really old voice recognition and TTS engines. Asobo has Azure which would make it sound really natural. Why not have it built-in option for ATC control and assigning commands in sim? That would be amazing not just for VR, but especially for VR.
Hand tracking needs hardware/driver support I guess, so it’s more difficult, but even with controllers it would be great to see your hand - with an arm, not a stump - I don’t know why everybody’s doing stumps. Whole body experience in modded Skyrim is amazingly immersive. Seeing your arms and legs and copilot in cockpit would only add to the immersion.
But it’s a good thing they are working on it…
@RomanDesign
I’d recommend you watch this video from the latest Game Stack Event:
PS: I’ve asked Martial on the Game Stack chat system whether they’re considering this, both for Voice Recognition (ATC for example) but also to supply a networked based and integrated system to voice coms (which to me should recreate aviation coms, i.e. one “lobby” per tuned frequency, instead of one lobby per group of players). He couldn’t say anything committing, but at least, that they’re looking into this.
One neat advantage is that PlayFab Party also does translation and text to speech, which means to me you could speak in French, and the automatic translation would impersonate you and speak in English to the other players.
It’s nice, but Asobo has all the Microsoft Assets and is already using MS Azure for TTS and Azure already has both text-to-speech and speech-to-text and is cloud-based, and supported by Windows. It’s already partially integrated in MSFS. So it would already make sense to go Microsoft all the way, to showcase the technology and don’t pay extra license fees, the way they went Bing for maps etc. They already have the solution at their disposal, they just not implemented it…
I’ve been wondering the same about another technology Microsoft acquired 4 years ago:
https://forums.flightsimulator.com/t/is-there-a-roadmap-for-the-terrain-lod-morphing-issues/385559/5
It does not work for many many languages. English speech I/O is getting shape now, but TTS as well as speech recognition is still emerging technology for non-English languages. That is ongoing development and it is slow. I’ve been involved for years. Languages like Dutch and French have a far larger lexicon and a different coarticulation. That makes TTS a completely seperate project for each language and recognition is also far more difficult. In the RW there are gadgets in Windows you can chat with. But avionics… you don’t want to start passenger boarding procedures at 20.000 feet resulting in decompression and crash, just because you’ve been misinterpreted…
For MSFS it should not matter. All ATC is in English all over the world, at least for IFR at large airports and control centers. Implementing that should be good enough, if you want realism and voice recognition operations. And if not - what’s already there should be good enough.
Even with English Only you’ll see foreign accents. Recognition must take these into account if you want everyone to communicate in English. Certain things are possible, like ATC/ATIS (barometer) requests that have a fixed format… channel change requests… maybe they can try to do these first, but I doubt if names could be supported with speech technology, or aircraft types, or exceptional situations… missed waypoints, change requests… confirmation of number sequences, in a foreign accent… the recognition side of this will need a lot of work ! Suppose you did all the procedures and the system misinterpreted your designation. You take off, your AP makes a turn to the south and you don’t know why… oops… Atrecht or Utrecht… small detail… opposite direction.
Well, I think all of what you said is already supported by Pilot2ATC software. I did try it and it works, and I do have a foreign accent 
And Pilot2ATC only uses the Windows built-in speech recognition and TTS engine, no fancy AI cloud processing Azure has. I made fully voice-activated IFR flights with it on X-Plane. See my video - it works fine. I could interact with it, initiate procedures, request different flight levels, headings, IFR/SIDS/STARs, vectoring, ground and tower clearances, runway holds, taxi instructions, basically everything. It even sees traffic and alerts you and commands a go-around if runway is occupied by traffic etc. I didn’t do readbacks because of difficulty writing down and reading back in VR (your “copilot” can handle readbacks and tune radios, it’s an option). But people are doing readbacks themselves apparently. And it’s easier when you see ATC window so you know what to say back. And all this with old technology. So there is no problem whatsoever using more advanced cloud-driven Azure to do even better job.
1 Like
Well the world is already moving on apace and Quest users can now fly in VR untethered thanks to Air Link. Personally I agree with everything the OP says but would urge Asobo to go one step further and implement hand tracking.
Currently I fly X-Plane VR with a Quest 2. I mainly fly gliders with a center stick and like the OP, I can fly with one hand on the physical stick and one controller in the other hand for flicking switches and turning dials. Now think about it - With hand tracking, we could adjust the seat position (actually can you? I know I can in X-Plane using Planemaker) but we could adjust the seat position so that when our hand reaches out and touches the virtual stick or yoke it actually touches the real stick or yoke, so we easily get the best of both worlds without anything dangling from our wrists. We can fly easily with a real stick whilst reaching out with our free hand to maniplate the knobs, levers and switches.
That’s true, but I think hand tracking needs a strong internal processor. That’s why it’s only available for developers on Quest, because it has enough CPU power to process its camera feeds and calculate hand tracking. On other headsets I don’t even thing camera feed is available to work with and it may be too low-res to work for hand tracking. So I’m not optimistic for that on G2, for example. LucidVR DIY haptic glove is much more possible, provided G2 works with SteamVR OpenXR well, which it doesn’t now. Controller would be strapped to the glove, and although cumbersome, your fingers would be free to for the yoke or operating cockpit stuff.
I find hand tracking using the Valve Index camera working quite good. Maybe not to the level of Oculus yet, but this is more an AI problem than a camera problem for now.
That’s good to know, I didn’t know Index has tracking. So there’s hope then…
Not Index tracking, but you can use Vive hand tracking SDK with the index cameras…
Has there been any word or hints on expected release date for controller support?
Nope, not that I’m aware of. They’ve just continued to state that they have a team working on it. Hopefully sometime this summer though, but I’m not basing that on anything.
The Developer Snapshot says “Started” and ETA of “2021”, so all we know is this year. My guess is that they are concentrating hard on XBox release / DirectX12 move. So that will likely happen first in the Summer. However they are also likely to have a separate team working in VR, unaffected by XBox project, so who knows. They hinted on QA that it’s “going well” so it should happen at some point. I finally started flying in VR (as opposed to building airports) and dialed in my setting so it’s at least tolerable if not great. Operating G1000/G3000 with a mouse (I have a trackball actually at my VR rig) is a nightmare! Especially any encoders. It’s a torture to dial in anything. And the cursor is often misaligned - it only aligns well if I close the right eye LOL! So most of the time I’m having problems clicking even on large buttons. I can understand people who say “mouse is just fine!” even less now. For me it’s positively torture, even regardless of the fact that I know how easier it CAN be with a controller…
1 Like
It would be a dream if Asobo will support hand tracking at least for hardware that is able to handle it (I.e. Quest).
It would be exceptional to fly the A320 and move the hands from the physical yoke to the virtual knobs and rotate, push and pull them.
I use the controllers with x-plane, but having to grip the controller and the missing of precision of the gesture to push vs pull a knob is a real pain and destroys immersion… hand tracking would be a complete different story!
Although remember that in X-Plane you feel the buttons click, encoders rotate and click etc. And what’s important, you feel the “snap” vibration when you grab anything or are at a button. With a hand you won’t know exactly when you are at the correct place to push or rotate something, and no feedback what you are pushing etc. With hand tracking you won’t have haptic feedback. So ideally there would be a glove that has a built-in vibration to let you know. DYI glove I referred to could be a good solution, it would have your controller strapped to you wrist (outward) with an offset in the software. But it needs the sim to work in SteamVR, which it doesn’t do well with WMR/G2.
1 Like