I’ve never noticed this personally, but what you’re likely seeing based on whatr you’re describing is the AP fighting you as you turn the physical yoke. When AP is engaged, the yoke in your virtual cockpit will be taking input from the AP over what you’re doing.
So if the plane is trying to go straight and you’re turning right, the AP will turn left to counter what you’re doing.