RollingCache.ccc performance debugging and tuning … How?

nenenui · February 6, 2025, 6:48pm

2025.02.05 - Comments to the Rolling Cache related information of the Dev Stream

Allow me to preface this post with the following:

IMHO it is extremely unfair to expect that a person can answer every question or knows every detail about a large project or topic
- … on the spot
- … in a live stream.
It is simply not possible to cover every technical detail
- … in short common language sentences
- … that will or can be correctly understood by everybody.
Everything is clearer in hindsight … or in “slow motion”.
I do not know the FS2024 code
- … and, looking at it from the outside, I may interpret some aspects totally wrong.

With that background I want to turn now to the Rolling Cache related information that was given in the last Developer Live Stream.

I downloaded the video from the link above and will provide the quotes with timestamps (h:m:s) … plus my comments. Where possible I will add links to some of my older posts where you can find more details and the background of my comments.

And as mentioned in many previous posts, my experience is only based on flying at 4K-Ultra-everything on PC. I know nothing about Xbox.

Data volume aspects

Let me start with the topic of data volumes:

01:19:10

S: "… the first hour of flying I downloaded eight gigs
    but then it sort of settled on an 
	  average of two to three per hour."
		
S: "… if you go from TIN to TIN at low altitude 
    it's going to fill faster … "
	
01:23:33

S: "… the average on PC was five gigs an hour …"

I have measured the same numbers as Sebastian.

However …

I think one should put a real number to the TIN case.
- A 3 hour flight in a (heavy) TIN region like Florida
  - KMIA to KJAX … 150 ktas … 1,000 ft
  - … did add around 60 GB to my Rolling Cache.
- 1 hour “TIN” = 20 GB

Why is this important?

The magic of FS2024, and a focus of the intro video that we are watching on every launch of the sim for 3 minutes, is … flying at very low altidute.

None of the FS2024 challenges is: “Flying at 40,000 ft from KJFK to ZBAA”.

All challenges are about … flying low and fast.

So I would recommend to embrace the obvious …

Flying low and fast … in countryside TIN landscape.

The future of the global twin is TIN

I really love the vision that Jorg repeatedly puts on the table:

01:17:17

J: "… I always love that about the digital twin.
    This is only going to get better.
    There's more and more data."

… and before that he mentioned the goal to increase “countryside TIN”. Later Sebastian added:

01:32:24

S: "… so first of all there's the world data
    … which is three or so petabytes.
    Every time there is a new TIN it's more and more data …"

This is all correct and IMHO this is what makes FS2024 such an impressive tool for exploring our planet.

However …

Precisely because of this increasing volume of TIN data
- … and precisely because that is the future of FS2024
- … the caching system must become at least one order of magnitude better at “caching”.
  - It at least needs to distingush between “important” and “not important” data.

I would recommend to embrace some …

Ideas for a “Hotfix” release of the Rolling Cache … which have been posted here:

Rolling Cache only stores the large data?

01:21:33

S: "… we have really two different caching systems.

    The rolling cache caches heavy stuff 
    like meshes, textures, and all these things.

    The little files, like CFG text files for planes,
    these are always cached on … the drive …"

Sebastian is clearly correct. There are two caching systems … actually I would argue that there are already three … and soon four:

Automatically managed downloads
- … cached (stored) in the RollingCache.ccc file.
- … stored (cached) in the normal packages on the filesystem.
Manually managed downloads
- … stored (cached) in the Community folder
- … and soon also FS2024 packages in some special FS2024 folder.

However …

The “heavy” vs “little” distinction seems somewhat oversimplified to me.
“Heavy” sound files are stored in the regular filesystem
- … and I suspect that this is for reasons of reliable data streaming to the sound subsystem of the OS.
There actually are lots of “little” files in the RC.
- I would argue that 40% of the files in the RC are “little”
  - … but, just to be clear, I am not saying that this is “bad”.

To back up my counterposition let me show concrete numbers. Over 6 flights with a total of 26 hours my 256 GB RC did archive around 5.5 million blob items with a total of 180 GB. A “rule of thumb” analysis of the distribution does look something like this:

Size category	Item %	Size %
0 to 1 KB	< 5	0.1
1 to 10 KB	35	5
10 to 100 KB	55	50
0.1 to 1 MB	5	30
1 to 10 MB	0.10	10
above 10 MB	0.01	< 5

Small items below 1 KB still use up 5% of the index area.
+50% of all items and blob volume is in the 10 to 100 KB category.
5% of the data volume requires 40% of the index (management) “attention”.

I did not (yet) make any similar analysis of the regular filesystem data, which is stored outside of the RollingCache.ccc file.

I would recommend to embrace this challenge:

The large number of small files does have a large impact on cache performance.

To repeat …

I am not recommending to remove the “little” files from the cache.
But I am recommending to raise the importance of the RC
- … so that it will always perform better than the filesystem or manual downloads.

RC size recommendations

During the stream a number of comments have been made with recommendations for the RC size:

01:22:10

S: "32 is what I usually use …

    If you put 128 gigs of rolling cache you're going to 
    have a very good experience because it's going to
    take … weeks to fill it up 

    … 32 or 64 is really … already very good.

    It depends how you fly …"

Again, I agree with Sebastian and in this thread I basically made similar recommendations.

However …

The essence of that post was:

Regarding the RC size we find ourselves between “a rock and a hard place”.

16 GB is too small (= big LRU problem) … but at least it keeps the index small.

256 GB “solves” the LRU problem … but the large index will cause “stutters”.

As it seems even a heavily used 16 GB cache may trigger more stutters than a clean new 16 GB cache (I have seen people with such observations).

There are a number of previous posts which explain the technical background of the “LRU” and “full index area dump” problems. You should be able to find them if you search for:

Todays Least Recently Used (LRU) strategy does not fit the nature of FS2024.
The sim always writes (dumps) the entire index area … up to 512 MB

“100% Full” question

One question related to the RC that was asked was:

“Could the rolling cache display information on how full it is and what it contains? Perhaps in My Library.”

… and the direct answer was:

01:30:23

S: "… that's I think something we will write down on the backlog …"

I agree with Sebastian that this is currently not the most important (pressing) issue they have. In addition I would like to point out:

At some point every cache will reach its size limit and be “100% full”.
- That is normal.
Actually one should expect that the RC is always at 100%
- … and if not, then it is not a cache, but a waste of storage.

However …

I think the question was really targeted at the problems that the RC is causing today:
- like “FPS stutter” triggered by a fully (large) cache
  - … where one “manual hotfix” (on PC) is to delete and rebuild the cache once it reached a certain “fill level”.
I fully agree with the person who asked the question, that the RC
- … needs to become more “transparent” to the user
- … which also means to offer more “customization” options to better fit individual flight habits.

I would recommend to embrace …

Ideas of Rolling Cache customization (configs) and some “analysis” features (maybe in the Developer menu?)

… and I posted some detailed customization ideas here:

I should revisit the topic of a “100% Full” cache in my next post in more detail. But just briefly … from the perspective of a goose the (hard) engineering question is:

How to design a cache that will not degrade in performance … once it gets big or full?

Sim Update 2

That directly leads me to the information about sim update 2.

I was very glad to hear that the FS2024 team is seeing the data download and data caching topic as one of the focus points for sim update 2. I hope that some information that has (and perhaps will be) provided here in the forum turns out to be useful.

Jorg repeatedly said that they want to take it slow in order to do it right.

And as a goose I would recommend the same. There is no other way. They have to get this right.