I have already adjusted the lighting in the room. Black drapes on top of curtains to attenuate the contrast the sun light creates on the window. No ceiling lighting, use a powerful flashlight, in a far corner of the room in my back, to light the ceiling and create as diffuse of a lighting as possible for the entire room.
Things are getting better though, I experience a lot less drifting, and the sudden 90 degree flip of my VR view has almost completely disappeared. I don’t know what to attribute those improvements to, it is not perfect, but it’s much better. I am sure will get there with VR, I just have to be patient.