This is written here:
Known Issues (content):
- The sim’s performance may be poor with the Mixed Reality Portal running in the background when not in VR mode.
- Workaround: Close the Mixed Reality Portal when not flying in VR mode.
As for switching back and forth, beware, sometimes it fails re-entering back into VR. It is happening sometimes to me but in general I’m also changing OXR settings when I do and this could be the main reason.