I have the same issue in VR. Sometimes it starts out at a proper zoom, but then somehow changes to a zoom greater than 1.0, making everything appear larger.
The VR settings have a world scale that I changed to 1.0, but the VR zoom stays at 1.4. If I hit the B key on my Quest 3 controller I get a grid overlay with a zoom slider, but changing the slider to 1.0 does not do anything…