I get you, and I reach for the same, but I believe the best solution is both: controllers + throttles/button boxes. The problem with button boxes and hardware is that there’s only so much you can intuitively find and use without looking, while wearing VR HMD.
In my control box, similar to your solution, I concentrated 6 DYI dual encoders with click (operating just like the ones on Garmin units!), 8 buttons and 3-pos switch (that works as bank switch, potentially tripling the controls, but I only use 2 banks, mostly 1).
I’m also building a custom yoke/joystick on hall sensors, so will have to throw away Logitech throttle quadrant, so I’ll also build a custom throttle quadrant with 6 axis, Boeing style levers (with A/T disengage and GA buttons and 2 analog reverse lever axis) and a few switches. And I will also build an additional box with gear lever + several light and battery switches too. I’m planning to use all that, but even if it’s not too much - that’s about the limit of what I could possibly remember and reach while in VR. I highly doubt it that you can use 16 of your X-touch buttons in VR, you won’t be able to figure out which is which, just by touch. Even figuring out each of the 8 encoders in a row you are touching would be a challenge. BTW in my box 6 dual encoders have 4 distinctively different shapes, modeled after G1000, to help and identify which one you are touching.
What about G1000/G3000 buttons, 12 each? 6 menu buttons for GNS530/G1000 are covered by my box, but not a rows of under the screen buttons. What about the GNS750 and TMB versions that are fully touch screen controlled? I could only map 3 buttons, zoom and almost useless encoders in TMS and the like, on my box. What about dozens of other buttons and switches larger airplanes have? Even if all that could be mapped - and it can’t be, I checked - no way you could orient blindly in all that in VR.
Mouse is a nightmare with those things, and controllers will be sweet. Grabbing a controller from my lap is not more cumbersome than reaching for the mouse, probably much less so. Then it’s just reach and point right on what you’re seeing.
My idea about outside mounting is just an idea, I haven’t tested it. But even with the controller strapped to the inside of your palm allows you to operate throttle or help with the yoke - just not very neatly - the controller gets in the way a bit, but yo can still grab and move things, because you’re not holding the controller, it’s just, well, there. Still beats the mouse by a lot. Just see the votes to my post. It’s #1 at the VR snapshot for a reason.
So I’m planning to use the boxes while possible, and grab the controllers for more complicated things, like FMC programming, cold start, engine starts, manipulating charts, writing down ATC instructions, touch screen usage etc. While flying, take-off and landing, most operations would be doable with hardware.