Plane drifts left and right

If you set a heading there are very few forces that will prevent the aircraft from holding that heading.
If you use NAV instead you are now effectively trying to follow a track drawn on the ground. Wind shifts, aircraft imbalance, thrust imbalance can all conspire to move the aircraft a little off that line and the autopilot has to continuously make corrections to hold the TRACK.