Sometimes it happens like that. But it depends on how you setup your flight plan too. One thing to make sure is you need to have at least 5% deadzone on all your primary flight controls axis. This is to avoid your hardware feeding the sim with micro-inputs which will override the AP, making you go off course.
There’s also things to check like making sure the assists are all hard/true to life. Then the flight model should still be Modern, and not legacy.
I almost never had an instance where the aircraft lost its course in the middle of a flight. Usually it happens around the time I’m entering a STAR on approach. But even so, I’ve done some things to mitigate that like: I always make a gate-to-gate flight plan with assigned SIDs and STARs and Approaches from the start of the flight. That way there’s no need for the sim to change the flight plan, and it will just follows it all the way.
Even on the occassional path just changes during approach, for some reason while it does look like the aicraft goes off course from the flight path. It’s actually still flying towards the original waypoint, so whenever I came across this, I ignore it and just continues since it’s still going to the same waypoint anyway, even though the line changes and it looks like it’s going off course. I always record my full flight to my YouTube channel, so events like this are always recorded and logged in case someone or myself need to review them.