I too notice these dramatic FPS loss at times. One thing I noticed was during the issue I used Task Mgr and saw that the GPU went way down in utilization. The minute it returned to normal, it went back to 90% and butter smooth. Very odd indeed.
Pivot
i9-10900K, RTX2070, 32Gb Ram, SSD