Next time you get a chance to check, can you see in Dev Mode if you are CPU Main Thread limited when this happens? What kind of CPU do you have? What is your Terrain LOD currently set to?
This thread might be related, too.