Massive GPU Usage spikes in MSFS 2020

It looks like an issue with the VRAM usage. It jumps from 3GB to 7GB and quickly back to 3GB. This shouldn’t happen. As you look around textures are loaded and unloaded very quickly it seems, which causes the stutters.
Do you have texture pre-caching on low or something?