I totally get what you mean and tbh I thought the same thing. My internet is constantly over 500mbps when tested and I tried different servers but still the issue remains. I’ve tried playing at different times of the day and no difference. This all started after the first update since release. The smoothest it has been was the release version on the XB. This happens even when I first switch on the XB before it gets to heat up.
A good example is try flying out of MMMX which was silky smooth before.