Standard Bench Flight for Performance Comparison

User Support Hub Virtual Reality (VR)
1

One of the main issues with comparing all the user feedback in terms of performance issues and regressions is the flight conditions used. Depending on whether you’re flying low over photogrammetric cities, dense forests or 2k up in the air over plains will easily make 10…15 FPS difference. Different airplanes and weather conditions as well as lighting conditions (time of day, season) add to that. That’s why I’m doing my performance analysis using the Flight Recorder addon with a pre-recorded flight using the same aircraft, same year, same date, same time of the day, same weather.

I suggest that to be able to really compare performance between different hardware systems and settings, we should use a standardized bench flight. This way, all the small but significant variations caused by manual steering including altitude, heading, speed etc. will be avoided too. The procedure described below should not take you more than 5min to set up plus the time for the bench flight.

I’ve included a link to a test flight below. If you haven’t been using Flight Recorder yet, it’s as simple as:

  • load the app
  • load the bench flight
  • embark on a normal flight anywhere on the planet with the conditions given below
  • ALT-TAB to Flight Recorder and press Replay Flight
  • ALT-TAB back to MSFS
  • make sure MSFS has set the correct time of day
  • start benching

Please bear in mind that if you haven’t been flying in the area, you’ll likely experience somewhat lower performance on the first flight(s) due to new scenery loading.

Here are the settings / conditions I suggest using as a standard:

  • Aircraft: DA40NG
  • Flight date: May 1st, 2021, 12:00 AM, the region is Central Europe.
  • Flight conditions: Weather = clear sky, AI traffic = off, photogrammetry = on.
  • Benching duration: 6min (can be set in CapFrameX), the recorded flight itself is longer. ! After launching the recorded flight, allow 25s before starting frame capturing to avoid initial stutters !
  • Benching conditions: Cockpit view. Headset placed on table untouched, view centered. No user interaction with PC during capturing - have a coffee or beer.

I suggest capturing frames using CapFrameX. Install, launch, and after launching MSFS select “Flight Simulator” as input/source. Specify capturing duration (360s = 6min) and ALT-TAB back to MSFS. Frame capturing can then be started without switching windows using CapFrameX’s hotkey, which should be F11 by default. I suggest using Average / 1% / Min FPS for reporting.

I hope this idea will work out and I’m looking forward to your numbers .

Here’s the link, please see if it works for you:

2 Likes
2

I’ll start with first results which I posted in a separate thread, though these numbers were recorded with the DA40-TDI instead of DA40NG. It is a comparison of 11900K vs. 12900K on a RTX3090 for OXR/in-games resolution settings 100/100 and 200/50.

ADL_MSFS_1b
ADL_MSFS_1b1609×600 38.6 KB

For fulls specs and settings please see Alder Lake i9-12900k - #70 by DerKlausi

1 Like
3

Keep in mind, setting the headset on the table will not be the same view for everyone. Even if you recenter the vr view, some headsets rest tilted up/down compared to others, seeing more cockpit or seeing more of the window & sky. Those that fill the view with more cockpit will have higher FPS.

1 Like
4

Good point, thanks. I only have experience with the G2, so should it rather be sth. like “view aligned horizontally, recentered” then?

5

But…errm…wait…shouldn’t centering view do just that, independent of headset orientation? I mean if I wear it I could look in any direction, press recenter and I’ll have the viewing direction centered in the cockpit? I could as well place a book under the front part of the headset resting on the table, so it would be looking to the cockpit ceiling when I was looking straight out of the windows before, but as soon as I press center view I should be looking straight out of the window again despite the book still there!?

6

Yes, when you hit SPACE the headset is centered, despite of it´s location

1 Like
7

only xyz position and yaw axis. roll and pitch remain locked to the real world horizon.

8

It would also be helpful to have a common set of settings per GPU so I’ll throw these out there for the 3090. These are actually the latest settings recommended by VRFlightSimGuy in his latest Reverb G2 settings video.
Anti-Aliasing TAA
Terrain LOD 200
Off Screen Terrain Pre-Caching Ultra
Terrain Vector Data High
Buildings Med
Trees Med
Grass and Bushes Med
Objects LOD 100
Clouds High
Texture Resolution Ultra
Anisotropic Filtering 16x
Texture Supersampling 8x8
Texture Synthesis Ultra
Water Waves High
Shadow Maps 768
Terrain Shadows 256
Contact Shadows Med
Windshield Effects Ultra
Ambient Occlusion Med
Cubemap Reflections 256
Raymarched Reflections Med
Light Shafts Med
Bloom On
Glass Cockpit Refresh High
CPU 10850K 5.0 All core
GPU Strix 3090 PL%123 CC+45 MC+600
RAM 64GB DDR4 3600
HyperThreading Off
Virtualization (VMX) Off
Game Mode On
HAGS Off
Windows Version 11Pro (beta 22000.318.0)

Given all of that, I ran 3 separate runs using the flight recording from OP in the DA40NG with time/day/traffic settings. Here’s my results. I posted about my settings in the “50/200” forum post that I get phenominal results with 75/150 on my system. Same holds true here. The FPS of 75/100 is not much higher than 100/100 but it is way way smoother than 100/100 and way way sharper than 50/200… I daresay it is nearly as clear as 100/100.

image
image982×358 10.3 KB

2 Likes
9

You could negate this and make the benchmark cleaner by having Rolling Cache disabled

1 Like
10

Why test in gpu limit? Set lod 400 clouds low test again 12900k will have better fps then 11900k

11

I’ve found in the past that getting people to follow exact benchmarking procedures doesn’t work very well, they forget a setting or click one thing different and the results vary wildly and become misconstrued.

That and the next Sim Update/GPU Driver Update can drastically change results, hell even Windows updates can affect FPS so you end up chasing your tail forever more which becomes a bit frustrating.

Still on the bright side it is nice to compare certain things with people’s setups as long as the benchmarks are created at roughly the same time, although ideally we’d have a built-in benchmark in the Sim to make life easier :slightly_smiling_face:

12

I think the goal is not so much to produce 100% comparable results but to get some measurement basis with actual numbers and not just people’s feelings. Because those feelings vary wildly between “stutter-fest” and “butter-smooth”…

2 Likes
13

The intention was to compare at the settings I use by default, not for “synthetic” benchmarking. However, I will try the settings you suggested.

1 Like
14

IMHO - yes and no. I think disabling cache will make the results more prone to effects of the different network connections and server pings people are using. So I would hope that the cache smoothens these differences a bit. But I may as well be wrong about this.

15

I honestly don’t think that having the rolling cache on really negates this. After all it’s an I/O both at a network and storage level and therefore CPU, drive and network performance all become factors that might skew results.

At least with rolling cache off you can note what your network performance is and add that to your benchmark performance data.

If using a cached flight it would probably be a better approach to manually cache an area. Share that cache (if this is even possible) and fly a set route between 2 airports within that cached area using only the manually cached scenery data.

16

Hi @DerKlausi , could you repeat the test after SU7? General opinion seems to be, that performance got worse. But check the settings! For me, the update messed them up…

17

Already done two series of tests - please check here:

1 Like
18

Thank you for the links!! Have you also tried render scale 100/100?

19

The point in time, update version, system info (ie. driver version, windows update version, available system resources etc) and all of that seem like they matter if anyone ultimately wanted to consolidate this into a single dataset and normalize the results.

Is some of this data capture and methodology something that might lend itself towards an add-on that could conduct these tests in a more consistent manner (or at least flag inconsistencies in settings used) and pull these results into a database?

I’m not technical enough to pull it off, but I’m sure others are.

20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.