QA & Bug Fixing Process

Discussion Hub General Discussion
1

First, all of this is written out of a constructive mindset, where I value this product and want to see it in a state where it is working more or less as specified. I want to offer my skills in any way that I can to help this come to fruition.

Second, I worked in games development — specifically flight sims — first as a QA “Tester” and then as a Producer. I was responsible both for finding and testing bugs, maintaining bug reports and interfacing with the programming/art/audio teams to see that these bugs were addressed, in addition to other Producer responsibilities. I am intimately familiar with this process and the obstacles throughout that may or may not be involved in why bugs are “allowed” to ship in a Gold Mastered product.

That said, I’m really confused by the bugs I’m seeing in this software package. There will always be bugs and the kinds of bugs I’m seeing aren’t the “I’m using an obscure graphics card with a 3rd party driver and, on Wednesdays when I fly upside down over Telluride the sim CTD” type of bugs. The kind of bugs I’m running into and I’m being frustrated by (as are many other posters here) are the easy-to-find going-over-a-basic-software-functionality-checklist-to-see that-the-rudimentary-functions-of-the-sim-are-working kind of bugs.

Examples:
• Default out-of-the-box keyboard Profile F10 and F11 keys do not work as described
• “Next Pilot View” from the default view freezes the sim
• Airports having aircraft-crashing invisible obstacles on taxiways and runways
• Bush Trip progress saving bugs
• ATC IFR Cruise ceiling issues, whereby ATC assigns a cruise out of the range of an aircraft’s specification or the pilot cannot request a lower altitude successfully either by radio or changing the flight plan via on-board computer and must listen to ATC repeatedly request an expedited climb
• Broken or impossible to complete achievements
• Basic aircraft systems, even in non-study-level aircraft, that are supposed to work do not.
Examples:
• Taxiing the DA40 broken
• DA62 always overheating coolant temperature
• TBM Parking Brake warning alarm can’t be overridden
• Starter safety switch in Junkers JU 52 isn’t required to be turned on, as specified in the Start checklist, for the starters to operate
• Radio sounds in headset when avionics switch and/or radios are turned off
• Ammeters showing massive draws when master battery switch is off

In addition to these low-hanging-fruit types of bugs there is the serious issue of “bug regression” where bugs previously squashed make their way back into the sim in newer released versions. Having a proper bug regression testing methodology (along with solid version control) is the key to arresting these issues.

What is concerning for SU8 is there appears to be something at Asobo going on that is allowing all this to occur and why will SU8 be any different? Is it a baseline poor QA dept.? Is it a management that is allowing bugs through because of lack of staffing resources to address them? or the business side of the company is pushing for release dates before the product is ready? or a “kick the can down the road” type of approach? or any other type of internal issues that could be the reason.

What I don’t really understand about modern-day development is how all of this even gets by Microsoft. If I sent a release version of a game to Nintendo or Sega of America, they’d kick the thing back if it had bugs. Showing my age here, you couldn’t patch a cartridge before we had the Internet we have today, so bugs had to be addressed. It seems there is almost a laziness in developers today, because releasing a patch via the Internet to your entire user base is so much easier than mailing them a package of 3.5” floppy disks.

How can I help you here? I”m willing to invest my own time and money to sign up for the Xbox development program, so I can beta test for you. I can do the same on my PC, if needed.

I love this sim, I’m completely smitten by it. I want to be a part of the solution here. Please, Asobo, take me up on my offer to help.

13 Likes
2

And let me also add that I bought the Base edition of MSFS 2020. I want to buy the premium deluxe edition and, while I’m not really into the air races, I want to buy it to support Asobo and this sim.

However, I’m not going to spend a dime on any of these upgrades until the major obvious bugs are addressed.

I cannot be alone here in this mentality.

5 Likes
3

I completely agree with your statements. I too have extensive experience in testing and software and am now heading an R&D organization delivering our own very commercially software in a different market.

I will add a few other very obvious regressions and extremely visible and easily reproducible bugs that should not have been allowed:

  1. VR crashing on exit VR mode. This happened in the past, fixed and now regressed. It can’t be that it’s not covered by a test case now.
  2. VR menu buttons do not function.
  3. VR POC for the Velocopter is completely off.
  4. Obvious crashes with popular addons - PMDG’s CEO just appeared on the Asobo dev Q&A. A day later, his DC6 crashes on load with DX12.
    These are so obvious that SU7 should not have been released that way. There will be a patch in a few days anyway.
1 Like
4

This kind of issues and bugs are the result of the simulator’s complexity, not of poor QA. The history of the genre shows that the first couple of years of a new product are always like this, and this is more complex than most.

It baffles me that people compare this kind of software to games on cartridges, which were simpler in the extreme.

Incidentally, there’s nothing that removes credibility from a post more than calling developers “lazy.” In over 20 years working in the gaming industry (even if tangentially as press) I haven’t met a single “lazy” developer. Game development isn’t a job you seek if you’re lazy. Lazy people wouldn’t survive in this industry for a month.

5

Spectrum Holobyte’s Falcon 3.0 and expansion MiG 29 were not cartridge products. They were both highly complex military flight sims that released on a stack of 3.5” floppies for PC. Do not confuse early PC graphics with a simplistic software program.

The lazy comment is more of a statement regarding “upstairs” i.e. management that may feel it’s ok to release a buggy product because of the ease of patching via Internet and forced universal updating to all of the user base. I am not speaking of those who sleep in their cubicles during crunch (including yours truly), who have little to zero say in how the product is allowed to be released. The cartridge comment came as a point about how we were require to submit a bug-free product to the publisher who then checked said product themselves. I’m trying to understand what Microsoft actually does here during “cert” prior to releasing a version.

You’re a journalist. Do you have actual in-company development experience to draw upon or are you basing your comments on opinion? It feels like you’re just being combative with me without really considering what I’m trying to convey, which is disappointing because I respect you,

1 Like
6

The complexity of the software you name isn’t even remotely comparable with a modern simulator, and even less so with MSFS, which is by far the most complex on the market, considering the systems it puts together.

I talk with game developers daily, on and off the record, and have done so for 20 years. I’m aware of a lot of the issues developers face because they tell me. Quite simply. I’ve learned a ton simply by talking to people. That’s why a lot of people in my line of business go on to become producers in game development when they get tired of the daily grind.

I considered what you try to convey. It’s just based on an extremely oversimplified view of the subject matter.

“it seems easy to me so it must be easy. How can they miss those obvious bugs?!”

Software isn’t usually shipped with bugs for the simplistic reasons you believe. At times they crop up because within such a complex simulation with many more interacting moving parts, fixing something after a beta can break something else, and it’s not feasible to put builds in an endless cycle of betas, QA, and new builds. Things gotta ship at some point. Some bugs are also well noted, but simply not prioritized because the developer has bigger fish to fry.

This is a good video to start learning something about the issue.

MSFS is likely the most complex commercial gaming software on the market, which likely places it as the most complex software for sale to single customers in existence at the moment. Complexity means issues, and there is only so much you can do about it.

Your post feels a lot like it translates as “I know better than the developers” which is a common attitude among gamers (and even more so among simmers), but a very fallacious one. You really don’t.

7

I really feel like you cherry picked, perhaps, my writing for key ways I may have poorly worded something rather than understanding what I said.

I didn’t blame the QA dept. and I don’t need to watch a video to understand the process. I held and did the job. I was there. I worked for EA, one of the biggest software companies there is.

When I stated “QA” I, literally mean, the entire quality assurance process from the lowly tester, to the company making a conscious decision to release a buggy product for a myriad of potential reasons.

The QA dept. may be the best in the world, but if the leadership of the company is beholden to making a Christmas ship date, then that sort of thing can influence shipping a product with serious bugs still in it.

And that is just one of a ton of potential reasons.

Yes, MSFS 2020 is highly complex, but a basic going over of, again, basic things to ensure they function isn’t asking for an over-the-top approach.

Yes, the weather system is mega complex and difficult to get right.

Yes, the photogrammetry is complex and difficult to get right.

Yes, the streaming of world data is complex and difficult to get right.

However, none of that is what I brought up. I’m talking about, for instance, the functionality of the default keybinds and making sure they actually work. That sort of thing is not complex, inexcusable and, when broken, are really obvious bugs that bring the userbase mega frustration.

That is what I am talking about.

2 Likes
9

Do not direct posts to others. Keep the subject on MSFS and the topic. A reminder of the Code of Conduct on this forum.

10

I, honestly, feel really sad that you’ve got a rebuttal to everything I’m saying and have, essentially, discounted me and my experience of actually being in software development.

You’ve put me in a box.

I’m going to step away from this conversation, because it feels like you’ve got all the answers and I’m unable to make any sort of a statement that holds ground with you.

I’m really sorry about that, because as I said, I respect you.

4 Likes
11

I appreciate that you respect me, but I’d rather you respect this sim’s developers. They deserve it a lot more than I do.

And oversimplifying the issues they face with the idea that basically don’t know what they’re doing (but you do) is honestly not very respectful. Nothing of what you describe as “simple” is actually simple.

Your experience in QA is noted and appreciated. But that’s only one level of the issue, and likely done on much, much simpler software. I can’t think of a single game by EA that comes even close to this in complexity and interconnected systems.

Flight simulators are different beasts altogether, and this one innovates on so many fronts that it’s a different beast even compared to previous flight sims.

12

Like the “complexity” of PRESS ANY KEY TO CONTINUE? Aside from the initial idiocy of the logic (no keypress can cancel so why have it in the first place?), when they “removed” it they didn’t… they ADDED code to bypass that on regular startup. We know this because when updates don’t go exactly as planned, some users still get PRESS ANY KEY TO CONTINUE.

That’s poor QA, not complexity. There’s lots more examples. Ding ding ding…

6 Likes
13

Agree 100%. I cannot believe that ANY beta testing was done with ANY VR devices. I have been in computer service and service management (now retired but still active) since 1964, electronics since 1958 and have NEVER seen a software release this poor and it has been progressively getting worse. There is no good REASON (excuses are plentiful) for lists of ‘Things We Broke’ a.k.a. ‘Known Issues’. Your comments about CTD on exit from VR is ‘right on’. It was repaired once, how did you (Asobo) manage to break it again? Your comment about the VR menu buttons not working is ‘right on’. If you (Asobo) knew that it would not function for all that fly VR, why was this not repaired before release. And, if I may add one FEATURE to the list…Going to ‘PAUSE’ has brought back the crash of aircraft to terrain feature that we had early on. It was repaired back around SU3 (give or take) but is again available as a special SU7 feature. I too would gladly Beta Test because apparently those that are testing are not reporting, or Asobo is ignoring what is being reported. VERY POOR management, bluntly stating.

2 Likes
14

I agree with all of what you are saying, and your list of examples really brings the point across. I have worked with complex SW & HW systems for many years, dealt with testing and QA for all the time, and understand on a fundamental, professional level how modern SW development is done and what results can be expected.

And this ain’t it.

The simple things that are broken, and the simple things that get broken from one release to the next, really cannot be explained away. These are not the hard-to-get-right, fundamentally new technology things that could be expected to have some teething trouble. It is not even just flightsim related things that are broken where developer factual knowledge growth could be an issue. It is basic computer science things like download processes that break themselves, basic gaming things like unachievable achievements, and basic HMI things like mouse wheels that scroll in the wrong direction and a head tracker that doesn’t.

That clearly indicates the problem is at the highest management level – Jorg, Seb, Martial. Other problems they have had during the last year have been knowledge related, not understanding how simulated aircraft systems need to work, but I think they are gradually getting much better at that. The fundamental graphics and terrain issues are improving tremendously as well. Basic SW quality is where they currently have their biggest gap, and that is a relatively easy one to fix. Just make it a priority and it will get done. But actually do it, don’t just talk about it in a Q&A session, words are not relevant, results is what matters.

I have done my part for now, dozens of Zendesk tickets filed as best as my QA skills allow in terms of clear problem description and reproduction steps. Will get back to doing that if I see some meaningful steps in the right direction in terms of quality.

2 Likes
15

Fully agree and good written comments here. I am forced again not to play cause they broke trackIR ( again ). Every update gamebreaking stuff happens. Never seen this so bad in 20 years gaming industry…

17

As a Journalist not sure why you are name dropping all the time, mentionioning other companies this is about Asobo, and flight simming. Can we keep to the title please not EA…but Asobo about flight Simmimg…

18

Thanks for making that offer of your time and professional experience, but as yet another ignored Beta tester, your time would probably be totally wasted.

You might be far better utilized as helping to RUN the Beta testing program or the QC department, but the chances of you getting to do that is extremely slim. (even if that was what you were offering !!))

Coming from an Aerospace Software Development Environment, I cannot even start to relate to what I see happening in Games Development, but at least I can relate to your frustration.

2 Likes
19

OP-et-all = I am plank owner of this sim, having purchased it on the day it was allowed to purchase long before release. I in fact was selected to Beta Test, but could never get the game to download and install, and right there I should have figured it out, it was going to be a mess. In fact, I am a FS Pro, since using sims since FS-98. I have played several games, but always came back to FS’s. I do not need to blow up stuff to feel satisfied. I do however get a great deal of satisfaction, planning a flight, in a 747, executing that flight, and landing safely that plane. I do not care it’s a GAME, it’s as close to the real thing as I will ever get. Take NOTE, thousands and thousands of fellow simmers HERE, also do exactly what I do, and guess what, they as well as I are just as upset, about the poor progression of this game. We are now in the 16th month post released and the bugs, ATC/Navigation/Plane bugs, keyboard gotchas, flight control (peripherals) resets every session, the list goes on and on. But all that aside, MS/Asobo-community-we are fighting the same fight.

There is one (BIG) thing which is totally outside of MS/Asobo’s control, PC Hardware. It’s my contention, that at least thousands, possibly hundreds of thousands of "spotty, poorly or counterfeit” made PC components are and in PC’s trying to run this game. I suspect at least 40-50% of the bugs are not bugs at all but PC’s which have marginal memory, GPUs rated at XX level, but in fact, should not have left plant. I believe there are/is memory leaks in this game, but those are compounded by memory both system and GPU/card memory which is possibly marginal if not substandard. I submit, lots of software parameters are overstressing the components which are marginal, and creating issues where possibly the issue is way less than reported.

YES, download/distribution issues in this platform are at least in the top 10% of the issues, as my main complaint is the poor distribution method used. Are/is bugs present now, you bet, but I would imagine, 10-50% of all users have had at least one instance where they have been unable to download patches/updates successfully. For me, at least 3 reinstalls, due to bad delivery of game through store, and my PC is at (when game released) top 5% of specifications as recommended by MS.

I do not program, but I was in charge of testing/recommending Windows 2000 workstation release in my company which had 3000 employees, and my action plan, tackled issues and solved said issues long before we released said OS to the company. Like I have said, if this was Operating System, and MS was releasing to public/companies/PC manufacturers, well Apple would win. It is appreciated, that MS/Asobo is taking “the point to fix bugs, etc. before more fluff is released”, that however should have been the position from the beginning. Yes, it’s complex, possibly billions of lines of code, from a development platform spread from sun to moon. Ladies and Gentlemen, it’s got to get better soon, or all the strides in technology will not matter, the user base will leave. Had I not been able to solve SU7 downloading/updating issues, with assistance from forum, as well as past PC knowledge, I would have pulled this thing from My PC, and never, ever used it again. I have FSX and X-Plane on my system, and both of those work 1000% better than this sim does, and guess what, I can get the same satisfaction planning that 747 flight on those platforms as I attempt to get here, the difference is, both of those work from the start to the finish, without any CTD’s, memory leaks which affect performance, or ATC telling me 5 miles out to descend from my FL350 to 2500 feet and land on Runway 13L. Or send me well pas the programed destination selected in the world map.

That this game is a visually stunning work of art, does not help, when my plane has rolled over, and is spiraling down because the flight model is broken. Please, please, please, get this thing fixed, or many people will just give up and chalk up their purchase to a bad choice, and XP-12 will eat your lunch

2 Likes
20

Really, is it officially announced?

21

A question about Training and Discovery flights - I thought that these would occur at a fixed time rather than at the current UTC. I tried one of the training flights from Sedona earlier today and found that it was still dark in Arizona. I find the same thing for the discovery flights. I reported this after SU5, but I am still finding it and wonder if I am missing something. I don’t see a way to adjust the time, but I really don’t want to do a VFR cross-country training flight in the dark. Thoughts?

22

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.