RollingCache.ccc performance debugging and tuning … How?

nenenui · November 22, 2024, 8:41pm

I had this “problem” all the time … since 2020 launched … and with 2024 even more:

How does the RollingCache really work? … I mean … really

In FS2020 it was very quickly obvious that the cache mainly causes problems:

With sim updates it could cause hangs (e.g. due to asset conflicts)
In regular conditions it causes FPS reduction and stutter (on my system)

So the solution for me was to simply disable the cache and be a happy goose.

With FS2024 it is different. The RollingCache.ccc is now a fundamental part of a “good system”, as streaming is now a “first class citizen” for all assets.

So the concrete questions that I have are right now:

Is there a good resource somewhere which does a deep dive into the RollingCache.ccc (RC)?
- e.g. in the developer SDK documentation area
Did someone document (or reverse engineer) the “file format” of the RC?
Does the SDK offer special RC tools?
- Dump a cache hit-miss heuristic
- Dump a list of all items which are in the cache
- etc.
Are there any “hidden” config options or files which allow to assign space to different asset types?
- … aircraft, sat images, geometry, etc.
What is the FPS impact of a … say … 2 TB RC vs the 16 GB version?
- Given that today a single aircraft ca have 5 to 10 GB of data I feel like 16 GB is not “a lot”.
What Windows tools could be used to really understand the cache access?
- I am looking at Resource Monitor and while I do see lots of “read” I have never ever seen a “write” operation. Curious.

So I will leave it at this initial list. Any pointers are highly welcome.

PS: Specially with all those server problems right now I feel like having a “good” rolling cache strategy might be helpful for me and others.

High level summary of my tests

Practical ideas (recommendations?) to improve local caching

Increase the size of your RC as much as possible
- … but I might change my opinion on this, once I take a closer look at the “tiny purple line” at the beginning of the cache.
- Especially the landscape data on long flights can and will wipe out important sim data
  - … which then needs to be downloaded again if it only was in the RC.
- Sizes of up to 250 GB seem to be without noticable problems (so far).
- Sizes above 750 GB might be a waste of storage,
  - … as the sim might not be able to use it all (due to the nature of the RC index structure).
Once the feature gets added to FS2024, manually cache (aka. “Install”) as many relevant FS2024 Marketplace assets (aircraft, airports) as you can outside of the RC.
- This form of manually caching outside the RC will reliably reduce the need for network usage.

Observed download (RC write) volumes

The following has been measured at “Ultra” graphics settings:

A very large and very detailed aircraft, like the Boeing 737 Max by Asobo
- … will cause a download of 0.4 GB during the “Config UI” phase
- … and will need to download another 1 GB for all the high LOD features.
Test P
- Flying with 70 ktas in a TIN landscape …
  - at 15,000 ft will download around 0.3 GB in 10 minutes.
  - at 1,000 ft will download around 3 GB in 10 minutes.
  - This translates to a utilized network bandwidth of 5 to 50 Mbps.
  - Higher speeds will increase the volume accordingly.
Test C to F
- 5 GB or more have been written to the RC by visiting a large handmade international airport (e.g. KSFO).
  - A large part are the static aircraft
    - … and vehicles (cars, trucks, busses, …)
    - … and common objects (boxes, fences, people, …)

Technical RC aspects that could be improved

Addressing the following aspects does require (significant) code changes.

The Least Recently Used (LRU) cache strategy does not fit the nature of FS2024.
- LRU was designed for random data with no meta info.
- However, FS2024 has and can collect a lot of meta information about the data:
  - Usage history
  - The data category
  - Pilot preferences
  - Game mode
  - Current flight plan
  - etc.
- FS2024 could adopt other (additional) well known cache management options … e.g.
  - Data category specific caching
  - Layered (generational) caching
  - Predictive caching

Useful Windows Tools for analysis

Resource Monitor
- Comes preinstalled with Windows.
- Can filter system activity for the FlightSimulator2024.exe process.
- Provides a visual high level overview of the system usage (cpu, disk, network, memory) over the last minute.
Process Monitor
- Provided by Microsoft for free via the regular app store as part of the Sysinternals Tools.
- Performs a highly detailed analysis of almost all system activities
  - … by obtaining a full event recording for a certain period of time.
- Process Monitor - Sysinternals | Microsoft Learn

2024.12 - FS2024 v1.1.10.0 Observations

Here is a short summary of the more detailed findings which are provided below:

2024.12.03-14.45 - v1.1.10.0 - Switching between Archer and Joby 5 times

My first tests suggest that the RollingCache.ccc is efficient and effective
- read and writes happen in large batches and 80% of all reads come from the cache.
- But there is a lot of network traffic in the background in cases where a goose like me would not expect any.
It seems likely (after the second test) that simple switching between already (fully) cached aircraft
- … will trigger network traffic in the order of a few MB per switching action.

2024.12.03-20.26 - v1.1.10.0 - 20 min idle in Change Aircraft

Being idle in “Change Aircraft” is a fairly inactive state
- … with only very minor network activity.

2024.12.03-22.25 - v1.1.10.0 - Scrolling around in My Library for 10 min

In general it feels like there is a (severe?) lack of local caching during the “Marketplace” access.
- No local caching always means more stress for the backend databases and servers.
Too large thumbnails … means too much unnecessary data transfer.
The RollingCache.ccc is not used at all to cache data (e.g. images, metadata) from the marketplace
- … which results in a very “broken” experience during times where the servers are busy
- as everything needs to be downloaded over and over again … while scrolling up and down.
  - So if I could see a thumbnail a few minutes ago … it might be gone a little later.
The network usage is low … but mainly because the requests seem to be queueing up inside the client (sim)
- … and so they are not visible in the OS events.

2024.12.04-22.43 - v1.1.10.0 - Launch of game - in 3 minutes - under server stress

Out of the total 4.4 GB read during a launch only 0.046 GB are read over the network
- … so, besides the very first launch, it clearly is not the amount of data which is “streamed” that has an impact on the launch time.
A warmed up RollingCache.ccc will deliver around 10% (0.57 GB) of the necessary launch data
- … so a warm cache reduces the network usage from 580 MB to just 46 MB … which is around 90% reduction.
It is yet unclear what has the highest impact on the extreme differences in observed launch times.

2024.12.04-16.44 - v1.1.10.0 - Flying for 11 minutes around Lana’i - 3600 ft

The RollingCache.ccc does deliver, as promised, a dramatic reduction in necessary network traffic
- … when it comes to landscape information (ground textures, 3D meshes, etc.).
However, based purely on an educated guess (as I do not know the code),
- … it feels like the sim does not cache responses to landscape related queries
  - … like “which texture ID at this LOD should I display for that GPS location”?
- This would explain why we see 20 times more data read from the network then written to the RollingCache.ccc.
  - This does not smell like “data” but rather like “data about data” (query results).

2024.12.04-18.26 - v1.1.10.0 - Flying for 11 minutes around Lana’i - low alt

Many observations of the previous “3,600 ft” test also apply to this “low altitude” test.
Multiple low altitude flights are needed to really stream all the data for a region.
- This indicates that the backend (API or CDN) most likely is not “fast enough” yet.
The 11 minute low altitude flight needed about 1 GB of additional landscape downloads
- … and on average 3-times more data was read from the RollingCache.ccc per second.
Some recorded events are hard to interpret and it could either be the result of “bugs” or “features”.
- Maybe, and this is a big “maybe”, the FS2024 sim uses an IO subsystem, which is “priority and throughput” aware. Maybe.

2024.12.04-21.47 - v1.1.10.0 - Selecting uncached aircraft - 737 Max (Asobo)

A very large and very detailed aircraft, like the Boeing 737 Max by Asobo
- … will cause a download of 0.4 GB during the “Config UI” phase
- … and will need to download another 1 GB for all the high LOD features.
Aircraft specific sound is downloaded during the “Config UI” phase
- … and is written into a dedicated file … and does not go into the normal RollingCache.ccc.
- This might be because the OS sound APIs work better if a dedicated file handle can be provided? Unclear.

2024.12.11-14.42 - v1.2.7.0 - Cache size increase from 16 to 256 GB

The speed of a bulk data append to the RollingCache.ccc is mainly limited by the OS and the hardware.
- On my system I could see almost 160 MB/s write performance.
  - However, this test cannot predict the real cache performance during a flight. It mainly provides the “absolute best case” limit.
For an “order of magnitude” context here are some numbers from previous tests:
- up to 4 GB … need to be read during the walkaround of large ultra detailed aircraft (see test: 2024.12.04-21.47)
- 2 to 5 MB/s … 20 to 50 Mbps … landscape data are read from the cache in the case of a low altitude flight with 170 ktas (see test: 2024.12.04-18.26)

2024.12 - FS2024 v1.2.x Observations

Here is a short summary of the more detailed findings which are provided below:

2024.12.18-14.38 - v1.2.7.0 - KSFO short slow drone flight

A major custom build international airport like KSFO can easily write 6 GB and more into the cache.
- So with a 16 GB cache and an “ultra aircraft” flying between two “ultra airports”
  - … even a simple two-way flight might required the first airport to get streamed twice.
Slow download speeds can result in server request timeouts (failures)
- … which can cause any kind of asset to only be visible in a lower than expected level of detail (LOD).
The “healing process” for objects that have been cached in a “corrupt” (low LOD) state, is … well … unclear.
- Maybe one has to wait until the data is dropped from the cache
  - … which would become a problem in (desirable) very large RollingCache.ccc configurations.

2024.12.18-17.43 - v1.2.7.0 - Launch game … downloads the same 40 MB again and again

Around 40 MB of downloads related to 301 files stored inside the SceneryIndexes folder get downloaded on every sim launch
- … even when they are bit-identical to the files which already have be stored on the local drive.
- This seems to prolong the launch process by 1 to 2 minutes under “normal” conditions.
There seems to be a lack of “meta-index” files with checksums and versions … an “index-index”.
- With such a simple addition the sim launches might be a lot fast.

2024.12 - v1.2.7.0 - Cache usage pattern … Is it a trivial FIFO ring buffer? No.

The RollingCache.ccc is NOT operated as a trivial FIFO ring buffer … and this is very good news!
However, this high level view can not show exactly which strategy (algorithm) is used
- … but frequently used data (e.g. favorite aircraft) should remain in the cache.
If recently downloaded and used assets gets downloaded again in a short period of time
- … then the size of the cache is either simply too small (see also: Test 2024.12.18-14.38)
  - or there is some kind of “feature” (bug?) related to caching priorities or asset versioning.
Compression of the RollingCache.ccc only shows clear benefits while the cache is not fully utilized
- … as in those cases there is still a lot of easy to compress “zero-fill” data.
- Or to reverse the logic. If compressing does not show much benefit, then the cache has already been fully utilized once.

2024.12.23 - v1.2.8.0 - Can one-time landscape TIN data remove long-time active data from the cache? Yes.

Terminating FS2024 via Task Manager seems to result in a clean shutdown.
- The next launch of FS2024 was without any warnings or problems … which is very good.
The RollingCache.ccc is most likely managed by a strict Least Recently Used (LRU) cache replacement policy.
- This will result in the unnecessary (re)downloading of previously cached data
  - … which puts unnecessary stress on already stressed servers
  - … and can for example significantly increase the sim launch time.
My practical recommendations did not change … but I think that I can now back them up with the tests provided above:
- Increase the size of your RC as much as possible
  - … but I might change my opinion on this, once I take a closer look at the “tiny purple line” at the beginning of the cache.
- Manually cache (aka. “Install”) as many relevant FS2024 Marketplace assets (aircraft, airports) as you can outside of the RC.

2024.12.23 - v1.2.8.0 - Will a “failure” to cache 3D data infect the RC for subsequent flights? No.

Stressed servers can result in very very very slow or even failed asset downloads.
Truly “failed” asset downloads will not heal, no matter how long the camera looks at them
- … but since healing can take many minutes it will be hard to distingiush “slow” from “failed” in a normal flight.
The sim will remember the failed state … and it seems that state is also stored in the RC.
- Most likely the sim does give up “permanently” on “failed” assets at some point.
- This is bad news … especially for a sightseeing goose or the official “World Photographer” mode.
However after a new launch of the sim, there will be new attempts to load the failed assets
- … and under good conditions the proper 3D planet will become visible.
So the good news is, that failed assets inside the RollingCache.ccc file do not result in “permanent” 3D data infections.
- It is not necessary to delete the entire RC.

2024.12.31 - v1.2.8.0 - More observations about (not) healing “broken” objects … is a mixed story.

Getting a clear picture of the different kind of “broken” objects is tricky
- … especially because some conditions are hard to provoke reliably.
Some “broken” objects do heal … after waiting long enough.
- The missing TIN data in LA required 19 minutes to arrive.
Some “broken” objects do heal … after a new flight gets started.
- The broken rudder texture of a static aircraft was such an example.
Some “broken” objects do heal … but require a restart of the sim.
- The “red error pilot” was such a case.
Some “broken” objects do never heal … because they are “broken by design”.
- The different car examples fall in this category.
The good news is that so far I was not able to trigger a case where the “broken” object persisted inside the RollingCache.ccc file
- … and thereby would have persistent across multiple launches of the FlightSimulator2024.exe.
- So there is, obviously, a cache state representation inside the RAM
  … and that cache can hold on to “broken” objects. But this does not survive a restart.

2024.12.31 - v1.2.8.0 - Rolling Cache data volumes during a normal slow sightseeing flight

The network bandwidth recommendations from the FS2024 team seem plausible.
- With Ultra graphics settings I have seen a demand for 5 to 50 Mbps.
Flying with 70 ktas in a TIN landscape …
- at 15,000 ft will read around 0.3 GB in 10 minutes.
- at 1,000 ft will read around 3 GB in 10 minutes.
Loading into an airport, taxiing or approaching an airport … those are the demanding phases of a flight.
Even without the excessive TIN landscape scanning, a normal 4 hour flight will fill a fresh RollingCache.ccc
- … and at the default 16 GB size it will get overwhelmed,
  - … as long as it is governed by a Least Recently Used (LRU) cache replacement policy.

2025.01.03 - Ideas for a happy new Rolling Cache ecosystem

Todays Least Recently Used (LRU) strategy does not fit the nature of FS2024.
- LRU was designed for random data with no meta info.
FS2024 has and can collect a lot of meta information about the data:
- Usage history
- The data category
- Pilot preferences
- Game mode
- Current flight plan
- etc.
Manual caching (a.k.a. “installing a package”) is a possible hot fix
… but FS2024 should and could also provide way better on demand caching options … e.g.
- Data category specific caching
- Layered (generational) caching
- Predictive caching

2025.01.05 - v1.2.8.0 - A more detailed Rolling Cache data access analysis

Process Monitor does not “record” all events. Some events seem to get “lost”.
With 72 (or 80?) bytes per index item
- … and an index block size of 512 MB
- … the index can hold up to around 7,456,000 references to blob items.
With an average blob item oft perhaps 50,000 to 100,000 bytes (0,1 MB)
- a RollingCache.ccc file larger than 745 GB may run out of index space.
- So there is plenty of room to grow a single RC file.
  - However, the performance problems may then be more related to in RAM index lookup (search) codes
    - … or the periodic index “full? dump to disk” persistency strategy.
- Somewhat confusing is, that for many blob items the HEADER is (far) larger than the BODY.
  - This feels like a waste of memory.
  - And why is there a need for a 1 KB HEADER in the first place?
    - 1 KB * 500,000 items = 0,5 GB of “waste?” storage.
  - And there are lots of “strange” blobs with 32 bytes to BODY.
  - Hmmm. Many interesting questions.
The access does suggest a “Read after write” access pattern.
- This makes sense because the RC seems to serve as a way to decouple chunked (rainy) server downloads from (blue skies) sim data usage.

2025.01.08 - v1.2.8.0 - Rolling Cache read-to-write ratio

It is highly likely that network downloads are initially cached in memory … and not inside the RC.
- Only data intended for the RC is then written to the RC.
- Data intended for the regular filesystem is not buffered in the RC first.
The observed Read:Write ratio numbers of the RC fit the different test characteristics
- … and the Least Recently Used (LRU) cache replacement policy.
The RC remembers too much “useless” data and forgets too much “important” data.
- Landscape data is usually (naturally) rarely reused
  - … which results in “R:W ratio” number of 1.0 or below.
- Other categories of data (airports, aircraft, etc.) … can result in higher “R:W ratio” numbers
  - … but a RC with a small size and a LRU policy, will struggle to keep them in the RC.
- Especially the FS2024 speciality, flying low and enjoying the new ground details and bioms,
  - … will result in landscape wiping out important data that should stay in the RC.
Overall real use “R:W ratio” number for the RC are barely above 1.0.
- The RC mostly acts like a buffer (queue) and only rarely like a real cache.
The current RC design does not seem to take away significants load from the API, CDN or Origin servers.

2025.01.10 - Ideas for a “Hotfix” release of the Rolling Cache

A “Hotfix” cache update should respect key constraints:
1. Any local caching solution must be able to work at the 16 GB default cache size.
2. No changes to the LRU Cache code base … or the present CCC file structure.
A “Hotfix” cache should try to achieve the following key goals:
- Increase the cache Read:Write ratio … significantly.
- Avoid increased cache item lookup latency.
- Avoid too many necessary code change in the sim.
This could be achieved by splitting todays RC file into three cache files for different content categories:
- HighReuse = Scenery Index, Aircraft, Airports, …
- MediumReuse = Globe (landscape) low + med zoom levels, Challenges, Marketplace, …
- LowReuse or AllOther = Globe (landscape) high zoom levels, Live data, …
Multiple cache files would allow to reduce the index area size in each file.
Introducing a config section in the UserCfg.opt file during a “Hotfix” would …
- allow easy customzation for nerd pilots who want to tune their PC system.
- prepare the ground for a “Brave New World” cache future.

PretzelGoose · November 22, 2024, 8:45pm

I’m upping mine to 200 now, not sure if it will make a huge difference though. But worth a try. I’m on Xbox X

MATHEWS1014 · November 26, 2024, 5:01pm

Any difference??? On my Xbox S after some flying it freezes and CTD. I understand that on smaller aircraft’s like Cessna it’s better. MY CTD happens happened on bigger aircraft’s like ATR and inibuild 320 and all.

smitty9792 · November 26, 2024, 5:13pm

I have mine set at 64gb and still, when i load in a plane i have used recently, the interior textures take time to fully load in, so it might still only be caching scenery

PretzelGoose · November 26, 2024, 11:13pm

It’s significantly better now, but I’m not sure if that is due to the upped cache or the fixed the servers - I’m guessing it’s mostly the servers

nenenui · December 3, 2024, 1:44pm

So far the best “official” and free tool that I could find is:

Process Monitor
- Process Monitor - Sysinternals | Microsoft Learn
- can be installed directly from the Microsoft app store for Windows
- It is a part of the … Sysinternals Tools.

Process Monitor is good for:

Performing a highly detailed analysis of almost all system activities
… by obtaining a full event recording for a certain period of time of
- file (disk) usage
- network usage
- memory usage
… and then extract specific events by applying searches and filters.

Due to the high level of detail the recording can quick reach the order of one GB of data per one minute of recording.

I will stick with this tool until I find a better choice. Next I will try to show some concrete “performance” analysis.

Nikita · December 3, 2024, 2:28pm

Went up tp 64GB from 32 yesterday on Series X. Not sure it is helping. Make sure you have enough free memory on your Xbox before setting a very high Rolling Cache!

nenenui · December 3, 2024, 6:59pm

So after upgrading to FS2024 v1.1.10.0 and warming up the RollingCache.ccc over Hawai’i with 7 hours over 40 flights I did the following trivial test:

2024.12.03-14.45 - v1.1.10.0 - Switching between Archer and Joby 5 times

This test performed the following steps:

Stay in the “Change Aircraft” section of the “Free Flight” area
Move to the “Other” aircraft area
Then repeat this aircraft switch five times:
- Select the Archer Midnight
  - Click “Configure”
  - Click “Save and Back”
- Select the Joby S4
  - Click “Configure”
  - Click “Save and Back”

So basically I was switching between those two views:

The Process Monitor recording then showed the following network and disk access results:

Before I go to my observations I need to provide some more info on my present FS2024 config:

Live AI traffic … is disabled
Group Chat and Community flight features … are all disabled
Weather … is using the “Clear Skies” preset
RollingCache.ccc … has a size of 256 GB

The positive observations are:

Disk
- In total 2.1 GB are read
  - … out of which 1.7 GB (80%) come from the RollingCache.ccc file.
  - So the RollingCache.ccc does provide the majority of the needed 3D data.
- Basically all bytes which are written (31 MB) go into the RollingCache.ccc.
- RollingCache.ccc usage seems very efficient.
  - Writing 31 MB used only 41 write operations
    - … which translates to around 760 KB per write.
  - Reading 2.1 GB used only 32,057 read operations
    - … which translates to around 65 MB per read.
Network
- The 31 MB of data written to the RollingCache.ccc
  - … matches almost completely the data which is read from the most busy ….deploy.static.akamaitechnologies.com:https connection
- Compared to the 2.1 GB which are read from disk
  - … the 31 MB (0.03 GB) is only a tiny fraction.
  - So the RollingCache.ccc does reduce network usage significantly.

The “negative” (confusing?) observations are:

Why is there “so much” network activity at all?
- … after all, I did already select both aircraft prior to even starting this test
- … and I tried to deselect all “community” features
- … and I am inside a “HTML” config dialog
- Without the bulk data there are a (sub)total of around 1600 network packets send
  - … which translates to 334 kB send, for no “obvious” reason.
The sim feel very chatty … while basically doing “nothing”.
- Some channels are sending lots of TCP packets … with very little data.
- e.g. IPv6 server 2620:1ec:29:1::45:https
  - sends 9,409 bytes … in 137 packets … which is an average of 68 bytes per packet.
  - receives 15,028,110 bytes … in 2,834 packets … which is an average of 5 KB per packet.
    - since each “send” triggers 20 “receives” (= 100 KB) this is a fairly large response.

Just to make it absolutely clear … I am looking at the OS events on my PC and try to guess what is going on. There might be very good reasons for what I am seeing, and I am just not understanding what I am looking at. However, sending unnecessary requests to already stressed servers is never a good idea. So perhaps there is (a lot?) of room for optimization.

I already did notice that something is “strange” with the ….deploy.static.akamaitechnologies.com:https downloads. In a second test I only switched between the two aircraft twice … and there was more data downloaded from that server then in this “5 times switching” test. So my initial assumption that those downloads are related to the aircraft which I selected are most likely wrong.

To summerize this test for the TL;DR crowd:

My first tests suggest that the RollingCache.ccc is efficient and effective
… but there is a lot of network traffic in the background in cases where a goose like me would not expect any.

nenenui · December 4, 2024, 12:19pm

2024.12.03-20.26 - v1.1.10.0 - 20 min idle in Change Aircraft

In order to get a better feeling for the reason why aircraft switching still triggered a 31 MB network read and related RollingCache.ccc writes I performed the following test:

Stay in the “Change Aircraft” section of the “Free Flight” area
Move to the “Other” aircraft area
Stay on this page for 20 minutes … doing nothing.

One of my questions was:

In “Free Flight” one can see the 3D earth
- … and even in “Change Aircraft” it remains visible as a very blury background.
Now what if the “invisible” earth is still trying to pull in some data updates?
- e.g. for the “Live” weather, which however in my case was disable.

So basically I was looking at this screen for 20 minutes:

The Process Monitor recording then showed the following network and disk access results:

My FS2024 config remained as in the previous tests.

The positive observations are:

Files
- Basically no write operations
  - … besides the periodic (once per minute) recording of datausage.
- All reads come from the MUSIC.PC.PCK sound file, for the background sound.
- No activity of the RollingCache.ccc … which seems correct.
Network
- Very little network activity … over 20 minutes of being “idle”
  - A total of 1,617 sends … means 80 requests per minute … so around 1 per second.
    - Those could be “status heartbeat” messages.
  - Around 4 MB read … which “on average” would be a (status?) response size of 2.5 KB (per second).
- So it seems like there is no background update of the “Free Flight” 3D globe
  - … so no useless weather downloads or similar activity.

The “negative” (confusing?) observations are:

It now seems likely that simple switching between already (fully) cached aircraft
- … , which was my pervious test,
- … will trigger network traffic in the order of a few MB per switching action.
- This looks like unnecessary stress on already stressed servers … for no obvious benefit.

To summerize this test:

Being idle in “Change Aircraft” is a fairly inactive state
- … with only very minor network activity.

BegottenPoet228 · December 4, 2024, 12:25pm

Interesting observations. Thanks for doing the work.

ButtOwipoLIVE · December 4, 2024, 12:38pm

32Gb is max MSFS 2024 can use, any more space is not going to make a difference

NixonRedgrave · December 4, 2024, 12:50pm

Interesting findings. I have always wondered the same questions about the cache. It’s been one of the sim’s great mysteries.

Out of your 256 GB total, how much of it have you filled in your experimentation and/or sim use? Is there a way to see that information?

I keep wondering if that same stutter issue that drove us to turn it off will still be an issue for us in this generation of the sim.

nenenui · December 4, 2024, 1:38pm

You mean for the size of the RollingCache.ccc ?

That could be possible, but such a size limitation would still feel “not future proof” to me. Besides that I can not even see a technical reason for such a limitation.

UPDATE: During todays dev stream Sebastian mentioned that some people are using a RollingCache.ccc with a 100 GB … and he did not discourage people from doing that. So I would assume that there is not an intentional “hard coded” size limit.

nenenui · December 4, 2024, 1:42pm

That is a good question … and I have not yet been able to find a simple way to check that.

Maybe the cache gets zero-byte-prefilled. Then I might be able to compare used-to-zero(block) sizes.

But honestly I do hope that Asobo might add some debug option, maybe to the SDK?, to extract some cache stats.

Besides the “size used” I would especially like to learn about the hit-miss-ratio … and I would like to allocate specific percentages of my cache to certain asset types (aircraft, geometry, airports, marketplace, etc.)

nenenui · December 4, 2024, 2:48pm

2024.12.03-22.25 - v1.1.10.0 - Scrolling around in My Library for 10 min

In this test I wanted to explore those questions:

How “chatty” (network usage) is the access of the Marketplace?
- … which right now is limited to “My Library”.
How good is the RollingCache.ccc support inside the Marketplace?

For that I performed the following steps:

Wait for a time of the day that might be “busy” on the FS2024 server side.
Go to the “My Library” section inside the “Marketplace”
Then repeat for 10 minutes the following tasks:
- Select the “Not installed” category
  - Scroll up and down a couple of times (in my 109 items).
- Select the “Streamed” category
  - Scroll up and down a couple of times.

So basically I was scrolling around in this screen for 10 minutes:

The Process Monitor recording then showed the following network and file access results:

My FS2024 config remained as in the previous tests.

The positive observations are:

Files
- The images for the official FS2024 packages are pre-cached on the local drive
  - … so they are read reliably.
Network
- Very little network activity … over 10 minutes of “scrolling”
  - … but … that is actually not a good sign (see below).

The “negative” (confusing?) observations are:

Files
- The images for the official FS2024 packages … read reliably from the local drive
  - but not quickly, as they are rather big
    - … 904x486 px with 100 to 400 KB is a pretty big “thumbnail”.
  - The open counters show how often each thumbnail was read from disk
    - … so it feels like there is almost no (?) caching in memory for such assets.
    - Which is why in total 54 MB have been read from disk … just to show a few tiny thumbnails.
- No activity of the RollingCache.ccc … which is not a good sign.
  - It feels like the HTML market interface does reload (all?) information from the servers
    - … after it (the image, the asset?) gets scrolled outside the viewing area
    - … data reloads over and over again, as I scroll up and down.
Network
- Very little network activity … over 10 minutes of “scrolling” …
  - which is mainly related to
    - A) the servers being busy and not responding to my requests
      - most thumbnails stayed black (data not available) during most scroll activities.
    - B) the client queuing the requests locally and not even trying to download thumbnails
      - … because most likely a maximum of already locally pending requests was reached.
  - So at a high level the marketplace was not able to deliver the tumbnails of 109 assets over a 10 minute period
    - … largely due to what a goose like me would consider a “self inflicted denial of service attack”.

The assumption that there is no caching would not even need a Process Monitor recording. It is easy to see. Sometimes I could see the thumbnail for an airport (e.g. the LGPA Paros), but after some scrolling or switching (e.g. to the “Streamed” area) it was “back to black” and no longer visible.

When I retested this usage pattern in a “everybody is asleep” time of the day, then the “My Library” usage did feel “somewhat OK”. It clearly was lagging and did not feel snappy … but I usually was able to see the thumbnails for my assets. But not so in stressful times.

To summerize this test:

In general it feels like there is a (severe?) lack of local caching during the “Marketplace” access.
- No local caching always means more stress for the backend databases and servers.
Too large thumbnails … means too much unnecessary data transfer.
The RollingCache.ccc is not used at all to cache data (e.g. images, metadata) from the marketplace
- … which results in a very “broken” experience during times where the servers are busy
- as everything needs to be downloaded over and over again … while scrolling up and down.
  - So if I could see a thumbnail a few minutes ago … it might be gone a little later.
The network usage is low … but mainly because the requests seem to be queuing up inside the client (sim)
- … and so they are not visible in the OS events.

ButtOwipoLIVE · December 4, 2024, 4:12pm

MSFS2024 | Rolling Cache Explained & What Setting is Best for You?

nenenui · December 5, 2024, 11:51am

Interesting.
This video so far seems to be the only attempt to explain technical aspects of the Rolling Cache … besides the remarks made by Asobo during the “Grand Canyon” preview talk.

However, there are a number of “pictures” and remarks which IMHO paint a somewhat misleading picture.

I have the impression that the term “Rolling” cache gets taken too literally in that video. I really really hope that FS2024 is not a using simple FIFO (first in first out) sort of ring buffer … but a more sophisticated cache logic (e.g. based on a Time-aware, least-recently-used TLRU policy).

Also the remarks related to searching inside the cache assume a very naive implementation with a linear search (a full cache scan) … while B-tree based (presorted) layered indices have been known for decades.

It seems like all in all it is still unclear what the technical characteristics of the Rolling Cache actually are … besides a very high level “boxes in boxes” diagram. So my hope is that over time we might get some better insight into this now very important component of FS2024.

For more background see:

FIFO vs LRU … Cache replacement policies - Wikipedia
Ring buffer … Circular buffer - Wikipedia
B-tree … B-tree - Wikipedia

nenenui · December 6, 2024, 10:40am

2024.12.04-22.43 - v1.1.10.0 - Launch of game - in 3 minutes - under server stress

After noticing during a flight that the servers seem to be under some level of stress, I wanted to explore those questions:

How long does it take to launch FS2024
- … in conditions where (some) backend servers seem to be under stress?
How much data does come from the RollingCache.ccc during launch
- … given that the cache has been already present during multiple previous lauches?

For that I performed the following steps:

Ensure that my PC system clock is in-sync with time.microsoft.com
- … to avoid the “self inflicted denial of service” bug which is already well documented.
Double-click the FS2024 launcher
… wait for the screen with the “yellow start button”
… click that button and wait for the main menu screen.

So the high level story is, that it took 3:03 m:s to get from a double-click to the main menu:

The Process Monitor recording then showed the following network and file access results. The first image has a focus on file events and the second on file write operations:

My FS2024 config remained as in all the previous tests:

Live AI traffic … is disabled.
Group Chat and Community flight features … are all disabled.
Weather … is using the “Clear Skies” preset.
RollingCache.ccc … has a size of 256 GB
- … and is already warmed up after multiple sim launches.

The positive observations are:

Files
- Out of the 4.4 GB read … only 578 MB came from the RollingCache.ccc
  - … which is roughly 10%
  - … and shows that most data comes from “normal” files.
  - Playing the intro PressStartVideo-4K.mp4 only is responsible 323 MB read.
- RollingCache.ccc in total is the busiest file (most events: 19,603)
  - … so per read around 29 KB of data have been extracted.
  - This looks like an efficient “block size”.
- The bulk of the data is read from many of the locally installed fsarchive files.
- Regarding the ca. 46 MB of bytes written
  - around 23 MB (50%) get written to the LocalCache\SceneryIndexes\fs-base-genericairports.dat
    - another 4 + 2 MB are related to glider and helipad locations.
    - It will be interesting to check, if that file is downloaded during every launch … and if “yes”, then “why”.
  - And there actually is around 5 MB of system log entries, which get written … to $LogFile.
    - It will be interesting to check what can be learned from those logs.
    - UPDATE 2024-12-10: I just realized that this is the low level “write-ahead log” of the NTFS file system. So nothing really specific to FS2024.
Network
- The amount of read bytes (46 MB = 0.046 GB) is almost the same as the bytes written to the files.
  - So to some degree those bytes get “cached” locally, which is good,
    - … but the question remains: Will those files will get download again on every launch … even if they have not changed?

Since the launched finished in around 3 minutes, there is nothing really “concerning” around here, however …

Sometimes I have recorded launches which finished in just 2 minutes
- … and some people report that they have even seen successful 1 minute launches.
So since local file access cannot explain such a difference
- … and a network transfer of just 46 MB also can (or should) not be the cause of an unpredictable 1 minute difference
- … it does seem very likely that stressed backend database servers or API middleware servers might be the reason.

To summerize this test:

Out of the total 4.4 GB read during a launch only 0.046 GB are read over the network
- … so, besides the very first launch, it clearly is not the amount of data which is “streamed” that has an impact on the launch time.
A warmed up RollingCache.ccc will deliver around 10% (0.57 GB) of the necessary launch data
- … so a warm cache reduces the network usage from 580 MB to just 46 MB … which is around 90% reduction.
It is yet unclear what has the highest impact on the extreme differences in observed launch times.

NixonRedgrave · December 6, 2024, 2:22pm

Do you have any purchased Marketplace content from 2020 — specifically airplanes since scenery has been disabled?

If so, could those account for loading time differences? It is unclear how additional packages are affecting load times.

(I’m fascinated by your process here.)

nenenui · December 6, 2024, 5:01pm

Good and important question.

I will add that to my “test todo list” as it is very plausible (given the FS2020 history) that having add-ons in my Community folder might (does?) add some (significant?) launch delay.

However, I have seen (recorded) a 2 minute and a 3 minute load time with exactly the same community folder addons (coming via Orbx and Contrail in my case). But a test with a truly empty community folder is needed, indeed.

PS: … and thank you for actually posting to this thread, as I discovered that the “forum bot” does not allow me to post more then 3 tests (messages) in a row … unless someone else also does a posting. So now this happy goose has “permission” for posting two more tests.