RollingCache.ccc performance debugging and tuning … How?

[nenenui]

2024.12.04-18.26 - v1.1.10.0 - Flying for 11 minutes around Lana’i - low alt

After the pervious 3,600 ft flight the obvious idea for the next test was: How about a low altitude flight?

• If I always fly the same short path …
  • … but this time very close to the ground to force the highest resolution of landscape rendering:
• How much data will now (have to) be loaded from the servers?
  • … in addition to what already is in the cache from the 3,600 ft round.
• How much data comes from the RollingCache.ccc instead of the network?
  • … especially after round zero, when the cache should already be warmed up.

For that I performed the following steps … which are almost identical to the previous test:

• I used the FS2024 “FCR” tool recording round zero to warm up the RollingCache.ccc.
• I then replayed the flight three times.
• I watched each flight from the same external camera position
  • … (hoping) to avoid too much data triggered by the glass cockpit
  • … and to cover more landscape then from a cockpit view.
• But this time my 11:54 m:s flight, circling the small island of Lana’i (PHJH), was
  • … with the Vision Jet at a slightly more moderate speed of around 170 ktas
  • … and so low to the ground that I was be able to see rocks, trees, grass etc.
  • … to trigger a very high (ground) LOD data demand.

So this is the camera view that I enjoyed for four times 724 seconds:

The Process Monitor recording then showed the following network and file access results. The first image has the data for round 0 (cache warmup) and the second for round 2:

My FS2024 config remained as in all the previous tests.

Similar to the last test I first want to compare the data from the 4 different round trips.

	Round 0	Round 1	Round 2	Round 3
Network - Send Bytes	2,205,740	2,501,717	838,938	695,372
Network - Received Bytes	321,821,099	284,884,657	2,565,739	19,981,048
Files - All - Read Bytes	2,373,614,906	2,548,535,279	3,282,319,517	2,492,714,947
Files - Sound - Read Bytes	521,012,880	521,012,880	900,418,196	521,012,880
Files - RC - Read Bytes	1,778,284,607	1,915,207,059	2,295,163,152	1,921,592,853
Files - All - Write Bytes	1,096,984,755	1,084,647,114	551,667	18,077,890
Files - RC - Write Bytes	1,096,980,019	1,084,644,155	547,147	18,074,177

On average (per second) over a 724 second flight the above translates to:

	Round 0	Round 1	Round 2	Round 3
Network - Send Bytes/s	3,046	3,455	1,158	960
Network - Received Bytes/s	444,504	393,487	3,543	27,598
Files - RC - Read Bytes	2,456,194	2,645,313	3,170,114	2,654,133
Files - RC - Write Bytes	1,515,165	1,498,127	755	24,964

For the analysis we now also have a baseline from the previous 3,600 ft test. Some aspects are identical to the 3,600 ft test:

• Files
  • All data written goes into the RollingCache.ccc.
  • Almost all data read comes either from the RollingCache.ccc or a local sound package.

The major differences to the 3,600 ft test are:

• Files
  • The amount of data read from the RollingCache.ccc is
    • around 700 to 900 KB/s … at 3,600 ft
    • around 2,500 to 3,200 KB/s … at “low altitude” … so a factor 3 more.
  • In “low altitude” there have been a lot less bytes read from the sound packages … a factor 2 to 3 difference.
• Network
  • At low altitude it required flying the same path more than two times
    • … before all data was streamed to the RollingCache.ccc.
  • Round 3 has reached almost the same level of bytes send as the “3,600 ft” tests.
    • So this suggests that most data, that needed to be cached, actually was downloaded by now.

Looking just at this “low altitude” test, the positive observations are:

• Files
  • All data written goes to the RollingCache.ccc
    • But why was 3 times more data written then downloaded?
      • Perhaps the rolling cache stores data in a preprocessed format?
        • This actually could be very good news.
        • If, and that is again a pure guess, if RollingCache.ccc uses the native in-memory sim engine data encoding, then bringing back data from the cache would be way faster, then the initial decoding of the streamed data.
      • However, “Network Bytes” are usually counting compressed data streams.
        • So maybe it just means that RollingCache.ccc data is simply uncompressed … and not preprocessed?
          • That would then be bad news, as it would mainly mean wasted disk capacity.
• Network
  • A higher LOD demand, due to lower altitude, requires more frequent updates of the landscape
    • … and so naturally the volume of data send and received will go up.
    • Here we see consistently a factor of around 4 to 5.
  • Round 0
    • Almost all downloaded data comes from “static” data servers (CDN).
    • This should be the high LOD ground textures and 3D mesh data, etc. So this is expected.

The “negative” (confusing? interesting?) observations are:

• Files
  • Round 1
    • We can see the same large amount of writes as in round 0 … a total of 1 GB.
      • This is an indicator, that a lot of landscape data was still not streamed in round 0.
        • This would point to slow or overloaded backend servers (API or CDN).
  • Round 2
    • Read bytes did go up by around 0.5 GB … which is very interesting
      • … as this goes hand in hand with a doubling of sound data read from disk
        • … and a radical reduction of bytes that needed to be written to disk.
      • In the “3,600 ft” test the demand for sound data was even higher
        • … however, the total sum of bytes read form files was practically identical … 2.5 MB/s.
      • So what could this all mean?
      • Maybe, and this is a big “maybe”, this is a hint that the FS2024 sim uses an IO subsystem which is “priority and throughput” aware.
        • If true, then that would be very good news for FS2024.
        • Think of it as an “auto LOD” feature, but just for file IO operations.
  • However, why are we seeing the same disk read of 2.5 MB/s … even when in each round new data gets streamed.
    • One might expect that with more data streamed for a certain region, more data would be read from cache on later rounds.
    • This could either be a “bug” or a “feature”, depending on that is really going on inside the sim. Hard to guess from the outside, and hard to explain in a few words.
• Network
  • Round 1
    • Almost identical data volumes as in round 0.
      • So there was still a lot of ground data that was not streamed, due to high latency of the request.
      • On average we can see a download of 400 KB/s (4 Mbps).
  • Round 2
    • The download of static data is reduced almost by a factor of 100.
      • So basically no new data for the cache did arrive in this round.
      • However, the send bytes are still higher then in round 3,
        • … and so this indicates that the servers have been asked, but did not deliver (in time) what was requested by the sim.
  • Round 3
    • The amount of downloaded static data went up by a factor 10 again
      • … and so the reduction in round 2 most like was due to (too?) high latency of the backend (API or CDN) servers.
      • Even after round zero to 2 there was still a small amount of data that was missing in the cache.

To summerize this test:

• Many observations of the previous “3,600 ft” test also apply to this “low altitude” test.
• Multiple low altitude flights are needed to really stream all the data for a region.
  • This indicates that the backend (API or CDN) most likely is not “fast enough” yet.
• The 11 minute low altitude flight needed about 1 GB of additional landscape downloads
  • … and on average 3-times more data was read from the RollingCache.ccc per second.
• Some recorded events are hard to interpret and it could either be the result of “bugs” or “features”.
  • Maybe, and this is a big “maybe”, the FS2024 sim uses an IO subsystem, which is “priority and throughput” aware. Maybe.