Xbox user also, no it wasn’t always this way. Only after SU6. Not fixed in SU7 either like all them other little bugs that are still there (I hear ya). This popping issue does not make my Series X feel next gen. In fact I haven’t seen this level of popping since the early days of The Witcher 3.
Long wait until SU8.