The rolling cache has little bearing on memory use.
It’s a tool to help people with slow internet connections. Whatever scenery is stored in the cache, still needs to be loaded into the RAM.
What tool did you use to check memory consumption? The task manager is insufficient for that. Try Windows Ressource Monitor or tools like ProcessExplorer.