As one of the first Testers on this Simulator and a Beta Tester in General for Other Simulators there has been a couple of Posts on the Testing Methodology or lack thereof.
*There needs to be some serious changes to the Way Asobo/Microsoft do things especially the Beta testing phase .
Lets Break Beta Testing Down to its goals and objectives.
Beta Testing is one of the Customer Validation methodologies to evaluate the level of customer satisfaction with the product by letting it to be validated by the end users, who actually use it, for over a period of time.
Product experience gained by the end users are asked for feedback on design, functionality, and usability and this helps in assessing the quality of the product.
Real People, Real Environment, Real Product are the three R’s of Beta Testing and the question that arises here in Beta Testing is “Do Customers like the Product and is it working as intended ?”.
But more Critically the Beta must have a Test Plan , ( This is where Asobo/Microsoft need help )
Beta Test Plan
Beta Test Plan can be written in many ways based on the extent to which it is performed.
Here I am listing out the common items for any Beta Test Plan to include:
Objective: Mention the objective of the project so as to why it is undergoing Beta Testing even after performing rigorous internal tests.
Scope: Clearly mention what are the areas to be tested and what is not to be tested. Also mention any specific data to be used for a particular feature.
Test Approach: Clearly mention whether the testing is exploratory, what to focus on – functionality, UI, response, etc. Mention the procedure to log bugs and also what all to provide a proof (Screenshots/Videos).
Schedule: Clearly specify the Start and End Dates with time, number of cycles, and duration per cycle.
Tools: Bug logging tool and its usage.
Feedback: Collecting Feedback and Evaluating methods.
Identify and review the Entry and Exit criteria.
When does Beta Testing Fail:
- No proper plan to execute
Here Asoba/Microsoft does well, at least in having test phase and that it will be tested.
- Poor test management.
Here Asoba/Microsoft does well also, A forum is created and the opting in and exiting is well managed
- Tight deadlines due to delays in previous phases.
No here its start to go wrong, Releasing a test build as the final build 10 days before release is tight.
- Released unstable product.
Again, there are instances where the test was completely broken, Reno for example Not a single user could use VR, Same with latest Beta for Sim update 7 , when a watermark in VR would make you nauseas and give missive head aches.
- An improper number of participants – too few or too many.
300 users should be OK, per platform and Asobo /Microsoft is increasing the pool again.
- Too short or too long test periods.
Current test are way to short, it does not give Asobo time to patch launch day critical bug.
- Ineffective tools.
Critical Bugs are not fixed , hot fixes released while in Beta, to improve release day experience to the broader public
The zendesk or now the XBox insider Feedback are ineffective for the reason below .
- No effective feedback management.
Bugs are not acknowledged or any feedback given, or even a follow up question asked on a specific bug reported,
There is 0 communication between Testers and the developers. This is essentially the problem.
A good Beta has at least the following:
- Alpha Testing or Preview build should be signed off.
- Product’s Beta version should be ready and launched.
- User Manuals, Known Issues list should be documented and must be kept ready to be published.
- Tools to capture bugs, feedback should be ready and usage documentation should be published.
- No Showstopper bugs in any of the platform.
- All Major bugs discovered in Beta Test phase should be fixed.
- Beta Summary Report.
- Beta Testing Sign Off.
Now here is the Biggest issue, the bugs with Planes, the mouse, VR and many other where reported the first day of the Beta, some are show stoppers, Not a single new build was released, I mean there are Bugs in the Release that have been introduced since SU 5 , that they have not acknowledged or even attempted to repair.
Until the users like a Beta product, it can never be considered as successful.
This is really the Game of the Year, but it can be better.