Asobo needs to pick up the pace in fixes/features for future patches, while keeping MSFS stable

Discussion Hub General Discussion
1

So credit is due where credit is due. Patch 1.9.5 was overall a good patch. It fixed the VFR map CTD bug as well as fixing other bugs, without introducing another show stopper bug.

However, the number of fixes for this patch was very few. It’s understandable that the fixes for this patch was few because my guess is, Asobo wanted to release a stable patch that didn’t break too many things. In the previous patches, Asobo fixed a lot of stuff but with those fixes, they broke a lot of stuff too, leading to a less stable and more buggy MSFS.

However, I don’t think Asobo can go at a snail’s pace in fixing things and adding new features like patch 1.9.5 for future patches. Otherwise, the next 10 years will be a very long 10 years.

Instead, I think Asobo can accomplish their goal of making a lot of fixes, and adding a lot of features, while keeping MSFS stable, if they make use of the beta testing team made up of volunteer players. The beta testing team made up of volunteer players will be able to catch the major bugs that Asobo’s internal testing team can’t catch. And if Asobo makes uses of a beta testing team this way, it will allow Asobo to carry out more fixes for patches, and add more features in patches too.

I believe Asobo is kickstarting a beta testing team for the VR patch release. But I hope Asobo keeps or even expands their beta testing team so a lot of progress can be made with future patches.

In addition, if Asobo does a monthly patch with new features and a lot of bug fixes, it’s okay if they delay their monthly patch by a week or a week and half if they detect a show stopper bug such as the VFR map CTD bug, sensitivity menu bug, A320 left engine bug, etc. The community will be fine with a one week delay as long as the patch released is more stable.

Finally, I think allocating time for a quick hotfix within one week after a major patch is released will help. This way, the community doesn’t have to wait too long if a show stopper bug is released into the major monthly patch.

5 Likes
2

Why not? It took 10 years to get the previous FS, FSX, to a level of maturity, and it only had 2-3 Service packs and one DLC. And that was with immense third party support.

We’re barely two months out of Launch Day, in arguably one of the worst years for normalcy in modern history.

This thing will look different a year from now. But it’s Wave 1 of what was previously a standalone delivered shrink-wrapped software. Totally different beast. But it needs time, period. Demanding a high cadence where there is a finite amount of money and resources, well, “Fast, Easy, Cheap” pick two out of the three as the classic Engineer’s Dilemma goes.

4 Likes
3

This is hardly a “snail’s pace”, when you take a second to consider the scale of this game and just how much needs to be uncovered to fix even the most trivial-looking bug.

Also, they’ve publicly set themselves deadlines and have kept to them so far. The update was released yesterday, as promised. What’s the problem?

1 Like
4

It appears to me, they don’t know how to do REGRESSION TESTING!

1 Like
5

I wonder if they have automated regression testing in place. For the benefit of doubt, let’s assume they do. If they did have regression testing in place, it was still a failure in that it didn’t detect the sensitivity menu bug from a few patches back!

That’s why they need a beta testing team made up of volunteer players. The beta testing team can help to root out bugs, allowing Asobo to speed up the pace of fixes and introduction of new features.

6

There isn’t a problem with yesterday’s update in terms of stability. It was nice and stable and didn’t have any major bugs.

There is a problem in the amount of work and changes done. At the current pace of changes for patch 1.9.5, we won’t be anywhere close to where we want to be after 1 year. There’s so many more bugs to fix and features to add. More needs to be done. Just check the other threads/replies on the forum of people who complain not enough was done in yesterday’s patch.

To get the best of both worlds where many more fixes can be done, along with the introduction of more features, a beta testing team can help with that.

7

Agreed it needs time. But at the same time, I think they can add more fixes & features for future monthly patches, provided they have a good testing scheme in place. By a good testing scheme, a beta testing team made up of players would help a lot.

8

The thing about Beta Testing is you really gotta do a good job screening the tester pool. We’re seeing all sorts of avoidable errors which compound the program’s challenges but are not related or attributable to FS - things like people ignoring Windows OS updates that are pending (!!!) and wondering why a program with Windows dependencies isn’t installing correctly.

You’d need to vet this group’s general knowledge of PC hardware and software, vet their configurations while ensuring they (hopefully) are now more representative of the wide range of hardware people are trying to run FS on, vet their broad-band configurations and network connectivity, all before you can even issue credentials to the test environment. Otherwise, you will get a lot of false positives coming out of this group, which may not help move the cadence forward. It could even, in the worst case, cause more latency as the devs start chasing ghost errors that could have been addressed with a bit more diligence on the end-user’s part.

I’m not saying UAT testing isn’t useful. I’m saying, with the limited amount of time, money and resources available, beta testing may not move this product’s maturity as fast as some may think (or in worse case, not at all).

1 Like
9

I proposed a best practices regimen in a wishlist. I don’t know how much traction it will get, but the feedback cycle to/from the devs isn’t working right now it seems.

10

So Asobo can screen the tester pool to find testers that fit the profile they want.

In addition, the submission of too many bugs that will overwhelm Asobo is kind of silly. For example, lead testers can be assigned in the beta test team that are in charge of submitting the bugs found, and then other testers under the lead testers can submit bugs to the lead testers. This can avoid duplication of issues so Asobo gets a nice concise and organized list. But lead testers in the beta team aren’t even needed if Asobo quickly sorts through the bugs submitted. If multiple bugs describing the same phenomenon are submitted by the beta testing team, that is a sign of a problem.

Your worries about “false positives” and developers chasing “ghost errors” is silly because it assumes there is no organization and no methodology to separate the wheat from the chaff in your system. Any software testing team worth its salt will be able to easily use methods to separate meaningful bug submissions and bug submissions that aren’t as meaningful. Heck, Asobo can even put a grade next to each tester over time so that they know which tester tends to give more valuable information and which tester gives useless information.

No disrespect to you, but your method of overseeing a beta testing group seems really lazy without any organization and very little thought put in. Proper oversight of a beta testing group would be organized, there would be a methodology and procedure every beta tester would use, and there would be mechanisms to separate the wheat from the chaff.

Besides, many major games out there have figured out how to harness the power of a beta testing team made up of players from the game. This isn’t a new idea. This is done by many other major games out there and those games can produce much more stable patches, with a huge number of fixes/features per patch. While MSFS is a complex game, it’s not the only complex game out there and other game companies with equally complex or more complex games have figured out how to use beta testers to their advantage.

1 Like
11

As I stated before, with limited resources, including money and hires, you would need to build a Beta test organization, assuming what oversaw the Alpha Test isn’t already dismantled.

I’m simply pointing out that Beta testing without resources to properly implement it will indeed end up being more detrimental than helpful.

Given resource constraints, what decisions drive the creation of a Beta Test team which draws resources away from addressing current bug fixing and long-term development? Where is the data that says this approach will accelerate bug-fixing? It may sound like it is, but until someone comes up with a data driven business case that shows “We have x amount of money and resources. We either partition it up to create/run/manage a Beta Test team and deliver this much positive fixes, or we continue to allocate funds and FTEs towards the plan to bug-fix per the roadmap.”

Edited to add: whatever the decision is, let’s hope it’s meaningfully derived. Because quite a few support approaches and decisions immediately after launch were more of the “Throw spaghetti against the wall and see what sticks” rather than data-driven.

12

Four patches in 8 weeks and one scenery update.

People pretending the sim hasn’t yet been patched.

3 Likes
13

And still almost nothing is better than after launch.
Update problems, systems are a mess, lighting everywhere etc. Every patch breaks something. Count doesnt matter, quality does.

But, maybe someday it will be more stable and they can develope new things.

14

Thank you. A good, well written analysis and good positive suggestions. Unfortunately Asobo got absolutely hammered yesterday by the community on the 1.9.5.0 patch discussion thread for, as far as I can tell, the error on the timing of fixing press any key. The rest of the patch worked well and created no further issues as far as i can tell, but this has been forgotten in the tantrums, wailing and gnashing of teeth that press any key seems to be able to generate. Some of the vitriol on there is little short of disgusting and does no one any favours.

I would like to think that the community has more maturity than this, but I fear not. That makes me think that whilst your ideas are great, the infantile amongst this community will create the hyperbole and pressure on Asobo such that they will struggle to take a sensible measured approach. All we can do is plead with the community to take a breath and support Asobo in what they are trying to do rather than p**ping on them constantly. Unless the level of vitriol reduces i fear that MS and Asobo could stop listening to the community. And who could blame them?

3 Likes
15

The results speak for itself. When the testing team at Asobo miss something like the Sensitivity menu completely not working a few patches back, that is a sign of failed testing.

I already mentioned you can be intelligent and smart with the resource constraints. For example, ask the beta testing team to organize themselves with beta testing team leaders submitting bugs only (or perhaps assign beta testing team leaders if asking them to organize themselves isn’t feasible).

The data that says this approach accelerates bug fixing and makes releases more stable are other large game companies that do this. Other game companies with more complex games than MSFS, but the release is more stable than MSFS, because they used a beta testing team.

16

These are interesting and very relevant points. MSFS is a complex ecosystem: the users pc, azure cloud, AI, all that data from many sources etc. One element going awry could impact the experience significantly. This also makes it harder to find the root cause of some bugs. It is also of course the key to its brilliance when it all comes together. Educating the user base on this is key (witness a lot of the nonsense on here about melted buildings when you zoom in to 20 feet from a photogrammetry building and how bad that looks, and claims of deliberate downgrading of graphics).

Finding a good beta test base will be key and getting one sufficiently diverse and representative of all user types well definitely be a challenge

17

Yup. If Asobo can put together a good beta test team, they will be able to really help Asobo root out the bugs before the patch is released.

18

Tbh, two patches where mostly patches to fix the other two patches, and the first patch was only a patch for game launching/installation problems. They lost a month of patching issues that could be prevented if they tested them properly first.

19

A hotfix within 1 week of a major patch release would be nice if there is a show stopping bug like the sensitivity menu bug, VFR map CTD bug, etc. Waiting 2 weeks for a hotfix seems to be a bit too long. I hope Asobo can speed up their hotfix to within 1 week of a patch release. If there aren’t any show stopping bugs for a patch, then the hotfix isn’t needed.

20

They’re busy with the Xbox version, testing, tuning for ok frames on the new box, etc.

PC version will get more attention when that’s done and they have an RC.