Your controllers may be fighting you. Use Windows Gaming Controller applet to check if the controllers need some sensitivity adjustment. My example below.
Everything should be centered up if you’re not touching the controls. No twitching, nothing.
Another possibility is check your control assignments and make sure none of them are double-bound to the same axis for example.