RollingCache.ccc performance debugging and tuning … How?

2024.12.09-20.46 - v1.1.10.0 - Walking around the Vision Jet at PHNY

While getting ready for a flight I noticed in the “Select aircraft” UI that the servers seem to be very slow again (e.g. thumbnails for liveries have been missing). So I figured it might be a good time for some more testing. Sadly I lost a very long Process Monitor recording, because that app did crash (too much data?). So what I cannot show is:

  • The problems of streaming an aircraft which I have used almost all the times.
    • Why was that data no longer in the RollingCache.ccc?
    • Was there an upgrade of the Vision Jet package? … I see v1.8.0 now. Not sure if the cached version was older.
    • Why does FS2024 not read the recorded state of the Vision Jet? … I always need to pick my livery again and again.
    • Was this because I looked at the Boeing 787 Max before? … so is the aircraft cache limited to 2 GB?
  • Lots of related interesting questions … but I have no recordings related to this.

So … by accident … what I could witness instead was a perfect “ground tile LOD mystery” event. This is related to a message I left here in the forum:

For the purpose of this story I will use the following, totally made up, LOD definitions:

  • LOD-1 … lowest level of detail (IMHO this is way below “Low” grade).
  • LOD-10 … highest level of detail (call it “Ultra” grade).

My FS2024 config remained as in all the previous tests. Here I mainly want to stress:

  • RollingCache.ccc is at 256 GB.
  • Graphics settings include:
    • “Off Screen Terrain Pre-Caching” = Ultra
    • “Dynamic Settings” = OFF … so no “Framerate Target”
Ground tile LOD mystery timeline
  • 18.30
    • “Start flight” with the Vision Jet at PHNY Lana’i
      • In the last days I have performed all my flights around this island (see the previous tests).
      • With a RollingCache.ccc of 256 GB I was expecting that every landscape data would already be in the cache.
  • 18.32
    • First “Bandwidth too low” warning … matches what I can see.
    • Even right under the feet of my avatar it looks like LOD-1 ground material.

  • 18.34
    • Static data from the CDN servers seems to be coming in at little under 1 MB/s
  • 18.36
    • My avatar is still standing on LOD-1 ground material.
  • 18.39
    • My avatar is now standing on LOD-2 or 3 ground material.
  • 18.40
    • My avatar finally is standing on LOD-10 ground material
    • … but starting from the middle of the screen the old LOD-3 data is still present.
      • In the following picture I put a pink marker line to show the LOD jump.

  • 18.45
    • Another “Bandwidth too low” warning … and the ground tile situation did not change.
  • 18.46
    • Some parked aircraft start popping in at the other end of the airport.
    • … as usual some are without landing gear.
      • (Which might be a LOD bug resulting from a 3D graphics exporting tool, that does not understand the visual importance of a “tiny landing gear”.)
  • 18.48
    • The ground tiles start looking “almost” LOD-10 complete
    • … but some tiles are “still resisting” (see pink marker)

  • 19.20
    • Until now I left the avatar in its position and just rotated the camera left and right to check the (lack of) progress.
  • 19.21
    • I switched to the “Pause” menu … which automatically radically changed the camera positon.
    • I jumped to the settings to check my graphics settings and data usage report.
      • Mainly I wanted to confirm that “Dynamic Settings” = OFF … and so that there really is no “Framerate Target” (= dynamic LOD adjustment).
  • 19.22
    • Returned to the avatar view … and my LOD-10 ground had already degraded to LOD-7 or 8 quality.
    • So lets wait for LOD-10 to reappear.
  • 19.24
    • There is still a lot of streaming … Resource Monitor sometimes shows up to 4 MB/s
    • … but the avatar is still standing on LOD-7 or 8 ground.

  • 19.25
    • Some LOD-10 tiles are popping back in … close to the avatar an the aircraft.
    • Less area then I enjoyed before … but better than nothing.

  • 19.47
    • I still can see in Resource Monitor some CDN downloads
    • … but no changes to the ground tiles.
  • 20.18
    • Still no ground tile changes.
    • So I decided to walk away from the aircraft, look at it from other places somewhat more in the distance
      • … and hope that this “forced” data change might “heal” the ground textures.
  • 20.31
    • Instead of using the “idle time” to heal the ground under the Vision Jet
      • the tiles jumped back to poorer LOD-6 or 7 … and stayed there.
  • 21.08
    • No real positive changes. But lots of interesting screenshots which did show perfect examples of “LOD inversion”.
      • Tiles closer to the camera have lower quality then further in the distance.
  • 21.39
    • Still no positive changes. But here is one of the intersting “LOD inversion” shots.
      • Some ground in the distance still shows the wheel rubber ground marks … which I consider a high LOD grade.
      • But very close to the camera it is … pixel mud.

So lets stop here. I think I made my case.

  • On my favorite airport, where everything should be in the cache by now,
  • … even after 2 hours of network and CDN activity
  • … not even the basic ground tiles are present.
  • But most interestingly … some ultra LOD-10 ground content basically “went missing” while watching it.

None of this can be explained by poor CDN performance or a too small Rolling Cache.ccc. Those issues are clearly inside some other parts of the data lookup, data caching and post-processing pipeline.

Related ProcMon recordings

As I already mentioned at the beginning. Due to a crash of the Process Monitor I lost the related full recording. But I started another recording session at 20.46 which produced this result at 21.17 … so it contains the data of around 30 minutes:

This time I will not go into too much detail (as there is not too much) but I rather want to make those quick observations:

  • Even in this 30 minute period, where nothing really did improve (but rather degreded),
    • … I could see around 100 MB of received bytes from the CDN
      • … which all have also reached the RollingCache.ccc.
  • Some API connections … like Azure server 20.42.182.110 … have been really busy
    • … around 1,000 packets send with around 1 KB of payload each
    • … but not visibile improvement to the scenary (ground tiles) … but rather regression.

So to summerize the most interesting aspects of this test:

  • There is obvious loss of (very) recently locally cached ground textures and similar data
    • … which cannot be caused by poor CDN latency or throughput.
  • Even waiting for hours at one camera location will not “heal” the ground tile problems.
  • And even with all the streamed and cached data … a scenery can degrade over time to lower quality.