I was experiencing such issue after using “Active Pause”. You mentioned you don’t use “Pause”. In my case no issue when using real “Pause” assigned to a button or key. But the “Active Pause” (left hand button in the row of buttons on screen) was causing the gyro drift issues.