I’m hoping someone with a background in software development/design/testing, or someone from Asobo can answer this. How does Asobo test the sim and the updates? More specifically, what settings are they enabling disabling in Windows and MSFS when they run the sim? Do they test different systems with different hardware and settings? Do they use the most up to date GPU drivers? Is everything just vanilla?
There are so many posts with differing opinions and so many simmers with different systems and settings.
I’m just curious as to their process, or what their process most likely is.
It would be really nice if Asobo shares some main guidelines about how to tweak pc/windows/network and etc. settings to get most possible performance. I know there are numerous of system configs however in general shape it is gonna be really nice. I believe there is nobody else other than they to assist us in terms of tweak the system in most efficient manner…
The only people who can answer this is ASOBO, and I doubt they will. From someone who has/does this sort of thing, you cannot test on all types of hardware, unless you are Apple, then maybe you can. Testing is usual done on the most usual systems, and not even always on intel and AMD but in many cases one. It’s usually the most prevalent to get the widest coverage with the least number of system. Testing gets expensive so you try to automate as much as you can, but realize every change, every cpu,ram, OS, video card combo could require you to test again. The numbers are not in your favor. I do not claim to know what ASOBO do, but it’s probably similar to many other companies, they look to us as users as an extension of the testers. Then fix as production bugs.
I can see that this thread is about to get closed down.
So, steering back on topic and away from anything that cuts too close to the bone.
Asobo has recently implemented community beta testing, and looking at the last couple of updates this seems to actually be working.
This method definitely exposes test code to a broader range of hardware and OS/driver configs, network differences and of course adds more eye’s and minds to the mix in order to spot things.
I think the criteria for beta testers is that you be in the MS/XBox ecosystem rather than steam but ultimately there isn’t any difference in the code - being within that ecosystem likely just makes it easier to distribute the beta code to the testers.
Keep your eyes open in the forums, they periodically ask for test participants.
Assuming testing is being done on different systems with different hardware, would you think those systems are updated to the latest possible BIOS, motherboard drivers, hardware drivers etc, as those updates are available? Or, do you think their systems are just as they were at release?
I’m a developer… modern VS2019 paradigm (service oriented, agile) has two test stages
Unit tests
Integration tests
The first one involves writing test code for relatively small parts: a procedure, or an input method… The second one is the real thing, it involves procedural tests of certain user requirements, for Asobo e.g. “rudder handling for GA aircraft” or “fix for waypoint reversal” or “new SDK menu option”. The integration tests are done using a debug compile (MSFS), the unit tests are done within the development IDE. Whenever the sources need to be changed, you submit corrections in a repository. A manager or team lead decides what changes are put in the master repository. When this repository is decided, and a new SU-release is planned, it goes to beta testers: non-programmers who do install and (more) integration tests, at random. The feedback is put into a final maintenance scrum, where small issues can be solved and put in the master repository. Larger issues are postponed, until the next release. Then, it gets a release version number and the program is built, using a build script. That build script also prepares the downloads for the end user.
Propably they make test (even if we can doubt about that), but not the right test, because they don’t have any sim background and experience and sim aspect is not their priority.
it’s being tested by the community in a closed beta. Part of the people were hand selected, part on first-come-first-serve basis.
So I would assume there’s a lot of different hardware configurations being tested, both with older bios/driver and new bios/drivers, since that’s up to the individual end user.
Sorry, but @KenG1175 is absolutely right: we are talking about an “entertainment product”. That doesn’t mean that there are no (automated) tests whatsoever, but that wasn’t the point: the point is that the target audience (“the client”) is a completely different one: gamers (yes: and that isn’t meant to be disrespectful of all those who take their hobby very seriously, or are even professional pilots!).
And that is a very different market than if you write a “proper” simulator for “real-world educational purposes”: because lifes are at stake (if you train pilots wrongly!). And those simulators get certified, so changing code is way more critical than, say, when you break some shader code in a game like FS2020, or some aircraft doesn’t behave like in the real world etc.
Most FFS level-D sims are just expensive procedure trainers and quite often aircraft handling considerable differs from the RW counterpart.
Some are even wrong at the edge of their performance envelope and pilots are trained wrong due to these errors and I’m not even talking about the 737 MCAS fiasco.
