I completely empathize with your frustrations. The odd thing is that it seems system-specific. I have better VR performance in 2024 with similar settings and am thoroughly enjoying low-&-slow flights with all the graphics improvements and displacement mapping (13700,4080,Q3/VD,Win11).
It makes me think something is putting the brakes on in your specific config.