That isn’t necessary, because I use all of those peripherals at once with my Xbox.
As @TheSevenflyer suggested, the sim doesn’t like it when there are multiple flight surface controls mapped to axis on multiple active devices at the same time.
In easier English, just make a new control mapping profile for your Xbox controller (and mouse, if it’s mapped in anyway to control surfaces) and ensure that you don’t have the thumbsticks and triggers mapped to control surfaces. You can still use the thumbsticks for camera movement, but unmap the rudders, ailerons, elevators, etc. and just have those axis assigned to the Hotas.
The other thing I’d do is make sure your Hotas isn’t inputting “rogue” inputs by checking the deadzone settings in the sensitivity settings section. What I’ve sometimes found, on my Hotas, is the twist grip (for rudder control) needs more of a deadzone to prevent inadvertent rudder inputs, for example. I’d see these rogue inputs even when I have the twist grip lock set to prevent the twist action while pedals are connected — I could still see the rudder axis twitching while moving the flightstick through its X&Y axis while observing the sensitivity sliders for the twist axis. Increasing the deadzone helped to cure that.
You might have something like that going on with one or more of your axis.
The positive takeaway here is that there are many of us on Xbox with similar peripherals using them successfully. There’s a solution to be found to cure what ails your setup.