I saw this as well. That’s a guarantee that FPS will drop (sometimes drastically.) Until they fix the DX12 memory allocation issues you need to do whatever it takes to reduce VRAM usage.
I can’t really help guide you, because my GPU has 24GB VRAM, and I’m not having the same problem. I did find this helpful guide to which settings affect the CPU and the GPU.