For me, the problem when exiting VR was that my VRAM was overloaded and windows then used shared vram, which then led to heavy fps drops. since i took some of the load off my game (reduced 2D resolution and tweaked things) i can get in and out of VR as much as i want with no problems!
Check your Taskmanager if you switch
(This is not my GPU, this is just an example) 
I myself have a 3080ti with 12GB VRAM and when my shared VRAM exceeds about 0.8 GB then the stuttering occurs. (DX11)