Nah use hardware for the yoke/stick and throttle quandrant. The VR controllers are for operating the rest of the plan systems…nav, autopilot, etc…its actually pretty immersive to reach for buttons and switches.
Some people never get used to it and never like it though. There is also the problem of your hardware/desk getting in the way. Anything at yoke level or below and you’re going to bump the desk reaching for it without a custom setup.