I agree, it affects immersion.
When a knob being turned MUST stay within your (VR) field of view, it is not realistic.
It seems like I can sometimes get away with a quick look elsewhere if I am holding the curser button down to rotate a knob, but not if I am using the scroll wheel to do the same. Most of the time though, if you lose sight of a knob, button, etc., you lose control of it. 
I always fly in VR. I voted. Thanks for bringing the issue up.