Since you’re not really looking directly at the stick when you’re flying, and there’s no actual feedback that you’re actually holding it, either visual or tactile, it sounds like something that at worst would be vaguely disconcerting in a few situations, say startup procedures, but is probably just not even apparent most of the time.
Adding controller support would probably help with the immersion, giving the body cues to accept the environment in which it’s placed, rather than just visual feedback which is prone to all kinds of illusions.
That said, there probably are some scaling issues with the model. It was discussed in another thread that the 3D model probably is slightly too large:
Is that difference enough to trip the brain into not believing it? Without any real world experience in a Corsair cockpit, or being able to actually see and use your hands, I still find this a difficult issue to understand.