RollingCache.ccc performance debugging and tuning … How?

nenenui · January 10, 2025, 11:05pm

2025.01.10 - Ideas for a “Hotfix” release of the Rolling Cache

I hope that I have by now provided enough background about the existing caching architecture of FS2024, and establish enough evidence for present problems, that ideas for better solutions will not be seen as unreasonable or unnecessary.

However at this point, especially for a quick cache “Hotfix”, I think that it is important to define some fundamental constraints:

Any local caching solution must be able to work at the 16 GB default cache size.
No changes to the LRU Cache code base … or the present CCC file structure.

I will try to present some “Brave New World” ideas in the future, but at this point especially contraint (2) is essential:

While not very efficient as a caching strategy … the current code:
- … does not causes crashes … so it is a robust well tested code base.
- … does work according to specification … even when strict LRU is not a perfect choice.

So what would I consider to be the key goals of a cache “Hotfix”:

Increase the cache Read:Write ratio … significantly.
- This will require to reduce the observed cache thrashing tendencies.
Avoid increased cache item lookup latency.
Avoid too many necessary code changes in the sim.
- This is related to code that uses the “LRU code” from point (2).

Introduce a config for Rolling Caches

The first idea would be to add a new config section to the UserCfg.opt file. Today I would describe the FS2024 RC system like this:

{RollingCaches
	Version 1.0
	DefaultBaseFolderPath "C:\..d8bbwe\LocalCache\"
	SizeTotalMinimumInGB 16
	{CacheFile.1
		Path "ROLLINGCACHE.CCC"
		SizeMinimumInGB 16
		SizeRelative 1.0
		ContentCategory All
	}
}

Introduce 3 new Caching Content categories

To increase the Read:Write ratio I would propose to introduce three new ContentCategory options:

H) HighReuse
M) MediumReuse
L) LowReuse = AllOther

A new config could then look like this:

{RollingCaches
	Version 2.0
	DefaultBaseFolderPath "C:\..d8bbwe\LocalCache\"
	SizeTotalMinimumInGB 16
	{CacheFile.1
		Path "RC_HIGH_REUSE.CCC"
		SizeMinimumInGB 8
		SizeRelative 0.5
		ContentCategory HighReuse
	}
	{CacheFile.2
		Path "RC_MEDIUM_REUSE.CCC"
		SizeMinimumInGB 2
		SizeRelative 0.25
		ContentCategory MediumReuse
	}
	{CacheFile.3
		Path "RC_LOW_REUSE.CCC"
		SizeMinimumInGB 4
		SizeMaximumInGB 32
		SizeRelative 0.25
		ContentCategory AllOther
	}
}

The user interface for resizing the caches in the Settings area could remain a simple “Total Gigabytes” slider.

The existing rule, that the cache size can only increase, should also remain in a “Hotfix”.

The math to calculate the new sizes would require a little code, where the SizeMinimumInGB and SizeMaximumInGB should have a higher priority than the SizeRelative factor. But that obviously is trivial.

In the above example we would get:

HighReuse = 8 GB
MediumReuse = 4 GB
LowReuse = AllOther = 4 GB

ContentCategory = HighReuse

It should be of little surprise that the actual “magic” needs to be in a wise definition of what is a HighReuse content category.

Because of the fundamental constraint (2) the underlying LRU cache files will not be able to provide us with information about real usage rates.

If I revisit the long category list of one of my previous posts (2025.01.03 - Ideas for a happy new Rolling Cache ecosystem):

… then I might want to use the following distinctions:

small … vs. … large data units
few … vs. … many data units
immutable … vs. … frequently updated
high reuse … vs. … no reuse likely
everybody needs it … vs. … individual needs
on every flight … vs. … on one or few flights

But the FS2024 backend (origin) does, most likely, not have or does not provide such information to the local caching system of the sim.

Besides that, splitting the same resources over multiple cache files will, in the existing system, result in higher cache lookup cost. So we would violate constraint (4).

So practically, for a “Hotfix”, we are left with a single, suboptimal option: We can only use the data category.

Beware … I actually do not even know if that is “trivial” today. But looking at the SimObjectPaths config I assume that there is some predictable structure in the HTTPS request URLs and the resulting virtual file system (VFS) paths.

With that I would define HighReuse as:

Scenery Index
Aircraft
- Thumbnails and “Metadata”
- All owned aircraft
Airports
- My (frequently or recently visited) airports
- All owned airports
- Metainfo … like parking locations, runway info, etc.
Vehicles and “ground stuff”
- Airport ground vehicles
- Regular city trucks, busses and cars
People
- My avatar
- Airport people

I would guess that the above might reach a Read:Write ratio of 100 or more.

ContentCategory = MediumReuse

In the MediumReuse I would see:

Aircraft
- Static aircraft
Airports
- All “autogenerated” airports
Ships
Career Mode
- My office
- People
  - My instructor
  - Passengers
Challenges
- Metainfo … like descriptions, path indicators, etc.
Marketplace
- Metainfo
- Thumbnails
- Showcase pictures
Globe (landscape)
- Textures
  - Low zoom levels … 0 to 6
    - … required for the high level globe view
  - Med zoom levels … 7 to 8
    - … common for free flight airport selection

I would guess that the above might reach a Read:Write ratio of 10 to 100.

Regarding the globe (landscape texture) data I would like outline the following assumptions.

Map sources are often using a zoom level approach, which for example can be seen in the OpenStreetMap (OSM) ecosystem. Technically it is also a form of LOD but I will call it a “zoom level” to avoid confusion.

If I take the globe in the FS2024 main menu or the initial free flight location selection I would take OSM zoom level 4 as the actual starting point. The following shows the maximum cache size requirements for real world sat image data in the OSM ecosystem for each zoom level:

Zoom level 4 = 16 MB
- You can identify countries.
Zoom level 5 = 62 MB
Zoom level 6 = 244 MB
- You can identify major cities.
Zoom level 7 = 770 MB
Zoom level 8 = 2700 MB
- You should be able to place most airport markers.
etc.

So, no surprise, as each zoom level adds factor 4 more map tiles, the average storage requirement goes up by factor 4 with each level.

I have no idea how the Bing sat image data compares, in size per tile, but the order of magnitude should be similar.

IMHO a maximum of 4 GB of sat image (up to zoom level 8) data can be considerd as MediumReuse landscape. One could even argue that zoom level 0 to 6 (around 300 MB) might qualify as HighReuse landscape.

ContentCategory = LowReuse

In the LowReuse category I would see … AllOther data. Basically this should be a form of “catch all” unless already cached somewhere else.

This would imply:

Globe (landscape)
- Textures
  - High zoom levels … 9 and up
    - … for actual flights at normal altitudes.
- Elevation or TIN data
Live data
- Weather
- Air traffic
- Peer player aircraft

I would guess that the above would operate at a Read:Write ratio 1 or below … which is what todays RollingCache.ccc file usually achieves during a flight.

Index tuning?

With 3 cache files instead of just 1 the total amount of index area storage would go up from 0.5 GB to 1.5 GB. So in total the combined blob area would shrink by 1 GB.

Now one could argue, that a dramatic increase in Read:Write ratio justifies such nominal reduction of usable storage, because it increases the hit rate.

However, it should also be very easy to reduce the index area size in such a “Hotfix” scenario. An index with 128 MB should be able to manage around 1.7 Mio blob items, per cache file.

Even with a fairly low average blob item size of just 20 KB that translates to cache files with a blob area size of up to 32 GB each. And that seems like a worst case to me, because as I posted previously, it seems like the real average is perhaps 50 to 100 KB per blob item. Especially in the HighReuse cache category I would expect blob items to be multiple orders of magnitude larger.

Reality check

Going back to the common ideas for caching strategies that I mentioned in previous posts …

Data category specific caching
Layered (generational) caching
Predictive caching

… the “Hotfix” ideas outlined above only touch the first concept in that list. The other two would require more brave or radical code changes.

And just to be very clear, I again need to repeat this: What I have written here is not an announcement, not a prediction, not “insider” talk. I have absolutely no clue what the FS2024 team is doing or planning.

I am just a goose with opinions and ideas … and obviously with nothing more interesting to do at this time of the year.

To summarize the ideas presented above:

A “Hotfix” cache update should respect key constraints:
1. Any local caching solution must be able to work at the 16 GB default cache size.
2. No changes to the LRU Cache code base … or the present CCC file structure.
A “Hotfix” cache should try to achieve the following key goals:
- Increase the cache Read:Write ratio … significantly.
- Avoid increased cache item lookup latency.
- Avoid too many necessary code change in the sim.
This could be achieved by splitting todays RC file into three cache files for different content categories:
- HighReuse = Scenery Index, Aircraft, Airports, …
- MediumReuse = Globe (landscape) low + med zoom levels, Challenges, Marketplace, …
- LowReuse or AllOther = Globe (landscape) high zoom levels, Live data, …
Multiple cache files would allow to reduce the index area size in each file.
Introducing a config section in the UserCfg.opt file during a “Hotfix” would …
- allow easy customzation for nerd pilots who want to tune their PC system.
- prepare the ground for a “Brave New World” cache future.