We need an official Benchmark-Scenario in MSFS

Reading trough the Beta-Forum we see different results when looking at performance, which is normal, of course. Everyone tests on a different system, in a different scenario. Some talk about performance improvement, some about regression. Some talk about DX12, some about DX11, some compare Native SU9 with DLSS in SU10… in short, it is a total mess and anything than meaningful.

Not even 1% of the users reporting about performance are really providing any evidence (like comparisons with SU9) of how they are testing and i tend to believe that a mojority of them dont even use a proper way of testing for performance. It seems many are just going for the “feel” or “stutters”.

So, i would vote for a benchmark-scenario that any user can use, like many games do in the graphics menu. This would be much better to have an actual coparison about improvements/regression after updates.

An easy way to make a benchmark is to pick one of the discovery flights because they’re always the same plane, same location, same weather, same time of day, etc. Then just record it with CapFrameX and compare.

4 Likes

Too much hassle for an average user. And if everyone tests a different scenario, we have the same problem…

A typical benchmark in games goes trough all of the effects, like explosions and stuff.

We need one official, generic scenario which stores the results.

10 Likes

I’m not really interested in comparing my results to someone else’s, because I already know where I would stand against someone with like specs. Right at the top.

2 Likes

Its not what i mean, its about that users can test against their older benchmarks, so we have a baseline after Simupdates.

8 Likes

Like I say, I already have CapFrameX for that. I’m not against the idea of an in-sim benchmark if the developers can make it quickly and get back to work fixing the sim.

1 Like

Thats cool, but 99% of users dont. And they dont do proper performance testing. Most dont even use an FPS counter or have an understanding of how to create a meaningful testing-scenario.

Some fly a Cub in the deserts and talk about great performance, others are testing in a mainthread-limited environment, some in a GPU limited environment. We need one meaningfful baseline that covers different typical scenarios that everyone can use and share the results. Only then, we have a baseline when we are reporting performance improvement or regression.

Right now, the performance threads are a ■■■■■■■■■■■ and anything else than meaningful and can give the developers a completely wrong indication.

15 Likes

Essentially “timedemo 1” for Quake, if anyone remembers that.

The demos would ideally fix the graphics configuration to known defaults, a bit like the demos for X-Plane, which override whatever you have configured yourself. That would ensure accurate comparisons could be drawn.

6 Likes

I do, and maybe that’s why I’m such an old curmudgeon :older_adult:

1 Like

Passion is understandable, but please don’t bypass or use profanity folks. Thanks.

3 Likes

I already commented something similar to OP yesterday, so voted for it. Betas should come with a set of test cases that all users can run using the same settings and the same enviromental conditions (weather, traffic and location). Is the only way to have a real comparable record of the results among different users/systems.

Cheers

1 Like

Same here … I see folk with 5xxx cpu’s, and 3080’s with worse performance than I have even when my only OC is PBO enabled. My spec really is nothing special except for CL14@3200 and a mesh case. 3800X, rtx3060(12GB) and stock cooling that never gets hot - Like you say people don’t tune MSFS properly and just settle on mediocre.

2 Likes

I agree. A simple click to benchmark would be very useful. Comparing results to previous versions and betas etc. It’s actually standard procedure in software QA.

1 Like

Make your own benchmark. Use replay tool in experimental options. I have made one and will record using default graphics settings and then compare with next version using same time and year and also rain preset.

That’s great for you.

But it doesn’t help everyone do a comparable benchmark, unless they all use the same flight Etc.

On top of which, having a saved flight is not going to be 100% accurate. For example, if there are changes made to the actual aircraft systems that may also have an effect.

The idea is for a simple click to run benchmark that is not affected by weather or aircraft systems etc.

Many games and other software have this.

4 Likes

Agree, Asobo could use the replay tool and make a benchmark and make us replay that with a benchmark button in graphics settings.

I made a benchmark for 9 minutes right now all graphics,traffic and data set to default. Will post a video how i did it. Then maybe Asobo could make something similar for all of us to use simple :slight_smile:

Klax
clear skies preset:

rain preset:

Added Min, avg and max on the photos. Interesting both had same avg and clear skies had lower minimum than rain.

Even if we have an official benchmark there is so many different locations in the world that may be unoptimized. Better to create an own where you usually fly and then optimize your settings for that place. I’m sure you will not fly where Asobo choose to make a benchmark. The settings you choose will only be optimized for that official benchmark location.

Here is my benchmark i will use to compare if performance has increased every update from now :slight_smile: Still processing into better quality. First benchmark starts at 1:58. second starts at 12:52

Sorry for bad flying, first time i was flying hornet. Wanted to choose a fast plane for benchamrk.

My next benchmark will be Ultra preset with t-lod 400.

I could share the replayfile but i can’t attach that filetype here and it’s not official :wink:

Will make one more replay with only camera movement on ground.

Ultra t-lod 400.

Rain preset:

Clear sky preset:

Only camera movement on ground at KLAX:
Default settings:

Ultra tlod 400:

System:

Intel(R) Core™ i9-10900 CPU @ 2.80GHz 2.81 GHz
RTX 2080 Super
32GB RAM

3 Likes

I like the idea of a standardized benchmark. However please note that in VR these overlays (RivaTuner?) do not work. It’s difficult to view performance statistics during flight in VR. I use CapFrameX to make a 1 minute capture and view the analysis afterwards.

Anyway, the point is that if somebody creates a standardized performance test, it would also be needed to standardize the capture method and the statistics that are reported along with game settings etc.

2 Likes

Well, Asobo has that FPS counter in dev mode. I’m sure they can record benchmark log of that if they want to.

Besides if you make a benchmarktest in VR it’s running by it self. Why do you need to use the VR while it running?

I’m not complaining about performance though. I think the performance for this sim is really good. I can tune the settings depending on where i’m flying. I can run it in 60FPS if i want to with my system. I bet those with better hardware can run it with higher settings than me.

Optimised settings for KLAX area rain (know i will have 60FPS mostly everywhere with this setting) If i use complex airliners i may need to accept lower FPS:

Min FPS=37FPS
Avg FPS=57FPS
Highes (VSYNC ON)=61FPS

same settings clear sky:

Not much of a benchmark if you are essentially removing vital sim systems like weather and planes. What if there was a problem with the weather implementation? It wouldn’t show up in your benchmarks making the whole thing worthless. It is one of those things that sounds like a good idea until you think it through.

Other games do have built in benchmarks but they also tend to be unrepresentative of actual gameplay as any conscientious benchmarker will tell you. They will freely admit the only time they resort to using them is in, for example, multiplayer which has an element of dynamism which makes it difficult to replicate scenarios. You may say - but…but…that’s why I suggested removing weather and planes. Well imagine creating a benchmark for a multiplayer shooter that removed soldiers and explosions. What would that tell you?

Anyone can benchmark their own system (and it is probably the best way) but it will only tell them about their own system and nothing else. I can’t explain why I get better frames with a 3070 than some people on here with 3090’s but, if the frame numbers they are posting here arent waffle, then I am. Time spent focusing on the causes of that, I suggest, would be far more productive.

They could create a benchmark just to appease the forum but I really struggle to see the actual benefit.

1 Like

I only see the benefit of using it to optimise the settings for that specific area they choose.