An Alpha Testers Open Letter to Asobo / Microsoft on Beta Testing

Discussion Hub General Discussion
1

As one of the first Testers on this Simulator and a Beta Tester in General for Other Simulators there has been a couple of Posts on the Testing Methodology or lack thereof.

*There needs to be some serious changes to the Way Asobo/Microsoft do things especially the Beta testing phase .

Lets Break Beta Testing Down to its goals and objectives.

Beta Testing is one of the Customer Validation methodologies to evaluate the level of customer satisfaction with the product by letting it to be validated by the end users, who actually use it, for over a period of time.

Product experience gained by the end users are asked for feedback on design, functionality, and usability and this helps in assessing the quality of the product.

Real People, Real Environment, Real Product are the three R’s of Beta Testing and the question that arises here in Beta Testing is “Do Customers like the Product and is it working as intended ?”.

But more Critically the Beta must have a Test Plan , ( This is where Asobo/Microsoft need help )

Beta Test Plan

Beta Test Plan can be written in many ways based on the extent to which it is performed.

Here I am listing out the common items for any Beta Test Plan to include:

  • Objective: Mention the objective of the project so as to why it is undergoing Beta Testing even after performing rigorous internal tests.

  • Scope: Clearly mention what are the areas to be tested and what is not to be tested. Also mention any specific data to be used for a particular feature.

  • Test Approach: Clearly mention whether the testing is exploratory, what to focus on – functionality, UI, response, etc. Mention the procedure to log bugs and also what all to provide a proof (Screenshots/Videos).

  • Schedule: Clearly specify the Start and End Dates with time, number of cycles, and duration per cycle.

  • Tools: Bug logging tool and its usage.

  • Feedback: Collecting Feedback and Evaluating methods.
    Identify and review the Entry and Exit criteria.

When does Beta Testing Fail:

  • No proper plan to execute

Here Asoba/Microsoft does well, at least in having test phase and that it will be tested.

  • Poor test management.

Here Asoba/Microsoft does well also, A forum is created and the opting in and exiting is well managed

  • Tight deadlines due to delays in previous phases.

No here its start to go wrong, Releasing a test build as the final build 10 days before release is tight.

  • Released unstable product.

Again, there are instances where the test was completely broken, Reno for example Not a single user could use VR, Same with latest Beta for Sim update 7 , when a watermark in VR would make you nauseas and give missive head aches.

  • An improper number of participants – too few or too many.

300 users should be OK, per platform and Asobo /Microsoft is increasing the pool again.

  • Too short or too long test periods.

Current test are way to short, it does not give Asobo time to patch launch day critical bug.

  • Ineffective tools.

Critical Bugs are not fixed , hot fixes released while in Beta, to improve release day experience to the broader public
The zendesk or now the XBox insider Feedback are ineffective for the reason below .

  • No effective feedback management.

Bugs are not acknowledged or any feedback given, or even a follow up question asked on a specific bug reported,

There is 0 communication between Testers and the developers. This is essentially the problem.

A good Beta has at least the following:

Entry Criteria

  • Alpha Testing or Preview build should be signed off.
  • Product’s Beta version should be ready and launched.
  • User Manuals, Known Issues list should be documented and must be kept ready to be published.
  • Tools to capture bugs, feedback should be ready and usage documentation should be published.

Exit Criteria

  • No Showstopper bugs in any of the platform.
  • All Major bugs discovered in Beta Test phase should be fixed.
  • Beta Summary Report.
  • Beta Testing Sign Off.

Now here is the Biggest issue, the bugs with Planes, the mouse, VR and many other where reported the first day of the Beta, some are show stoppers, Not a single new build was released, I mean there are Bugs in the Release that have been introduced since SU 5 , that they have not acknowledged or even attempted to repair.

Until the users like a Beta product, it can never be considered as successful.

This is really the Game of the Year, but it can be better.

39 Likes
2

The very fact they cancelled the “Flight” testing team for SU7 is all you need to know about the lack of preparation, alas their whole approach to beta testing has up until this moment been very weak indeed.

1 Like
3

A fantastic post BoeingPilot8795, I would love your input as to why you think the process is so terrible why little to nothing is done with beta testers responses and why the workflow and timing is broken. Is it a matter of putting a tester part to it all just to shut people up? Not having the resources available to actually act, I mean yes MS has money but that does not mean the guys at Asobo has much to play with, I say this because of the broken flight models, obvious bugs etc. when saying they have “real pilots” working with them… what is the reason for this mess?

4

Your excellent post summarizes the A to Z of basic testing, it is clearly obvious that the process is failing in many areas. While you explain how it should be, I would like to pinpoint where the current process is failing.
I think the key problems in internal and external beta testing are:

  • problematic (or missing) regression testing, i.e. re-running tests on things that already worked, to ensure your changes haven’t broken anything
  • lack of appetite to postpone releases, so that constructive beta testing feedback is given enough time to drive corrections.

I am 100% sure that if they improved on those two, it would eliminate a lot of causes for frustration amongst users.

4 Likes
5

Watch this video, it explains a lot.

6

Also I have been watching how guys test something like the Zibo mod in Xplane, they have a hunger for carefully trying things over and over a few times and then reporting back using a carefully put together little spreadsheet… the big big difference is that Zibo (the guy himself) makes changes very quickly when things are raised giving the whole group of chaps who do the testing for fun motivation and momentum.

2 Likes
7

Excellent post. For me it boils down to one thing. The fact they have committed to a release date before the test starts shows this isn’t a test. It’s a marketing/PR exercise.

4 Likes
8

Thank you for your Replies,

I think the one way to fix some of the problems in the Beta Testing Phase is to have transparent communications, for an example the free look mouse bug that was reported on SU 5 Beta and every Test after plus on Zen Desk and the feedback Hub, ( If you look at the Twitch Feed from last week ) Martial Bossard does not even know the existence of the Bug ,

The issue here is they don’t want to implement hot fixing to there Beta Run, many of the Bugs would have been fixed, instead everyone has to wait for the next update cycle, sometimes 2, this makes for a problem.

Imagine buying a PMDG 737 $100 usd , and the next update breaks it, will you be happy with a month for a fix from Asobo / All this can be greatly reduced with a better Beta system with Transparent Comunication between Testers and the developers.

6 Likes
9

A reminder of the NDA.
Guidelines & NDA - Microsoft Flight Simulator

This topic has been referred to the Community Managers.