Either your FPS is low or you are getting motion reprojection artifacting (which is more of a wobble than a stutter), both of which are most noticeable with side views. If the former, you can either lower your VR graphics settings or, as I do, just accept it. If the latter, try turning off motion reprojection, but that can then introduce stutter. I prefer stutter over wobble, so motion smoothing is turned off for me.
In other words, welcome to VR and chose your fix “poison” wisely 