Let's discuss BETA testing

As with most of my posts, I like to get some of the “givens” and contentious points off the table first.

As this discussion has the potential of breaching the Non Disclosure Agreements that many of you have signed, please let me set some parameters for this thread.

DO NOT discuss any aspects of any flights you have been involved in that were done under the NDA!

We know that the beta process has some flaws. We know that the beta process is an evolving entity. We know that, currently, (Steam users cannot participate and that Asobo is working to change that.)
{**EDIT: This has been fixed for this current beta. Steam users CAN participate now.}
What we, as a community, don’t seem to know, is how the process works.

There are two primary levels of testing. In house testing, done by employees of the developer, and public testing. In this discussion I would really like to steer clear of the in house testing. First, we have absolutely NO knowledge of how Asobo has that structured and secondly, guessing at their processes will only result in the inevitable spear chucking.

So, lets look at the public form of beta testing. In general, the idea is simple. The developer generates an update. They provide a compiled copy of that update to either a closed group or, as in this most recent case, the user base as a whole. We users then take the update for a spin and provide feedback to the developer.

I think we just covered pretty much the entire understanding that the majority of users have when talking about a beta test. That would be the impression I get from every beta I have been involved in. Equally obvious is that there are some very good beta testers among us.

Beta testing is not for everyone. There are a fixed number of bugs and or features that are the focus of every update. The primary purpose of running a public flight is to see if there are any NEW bugs that arise from the changes introduced by the update. Regressions, can appear that may have slipped through initial in-house testing. Testers are also tasked with confirming that those items marked fixed, are actually fixed.

Where this whole process gets bogged down and causes many to become frustrated is when users don’t understand the process or simply want to have a backdoor the register their own complaints. I would never discourage anyone from getting involved. Everyone IS welcome, but if you want to jump in, you must understand what is expected.

Betas are not an opportunity to pick apart the program. You can do that all day in the general forums. When a beta begins, a list of items is provided. These are the focus of the testing. Stay on the list. Posting known bugs that are not in focus, just clutters the feedback. There is a finite amount of time to test the things that need testing. Don’t waste the developers resources with out of focus rambles.

The testing forum category is for posting your test results and observations. Detail is required. “Me too” is not a useful post in the general forum. It is simply useless in a test. There is a template provided for feedback. Use it. Every time.

I know, everyone is an expert, but please, before making a test report, research the subject carefully. To test an aircraft, you really need to compare to the real aircraft. If you have no experience in that particular cockpit then your tests must be purely numerical.

To test a weather system’s accuracy requires a strong understanding of aviation meteorology and weather sciences. If you have only ever seen the top of a cloud from the passenger seat on an airline, you are not qualified to determine whether the formations are realistic.

Discussions, while sometimes productive and can help find solutions, must be kept on topic. Important anytime we are using the forum but absolutely essential in a test. You don’t need to post in a thread, just because you disagree/agree. Offering some new, tested, data will help the discussion move to a conclusion, but joining in to express an opinion must be left to the general forums.

Finally and most importantly, I believe, is to understand that, as with any scientific method, repeatability is an absolute must. If you are testing a specific feature/fix, track the details. EVERYTHING. If you determine there is a problem, you need to be able to communicate the details that allow others to duplicate your findings.

Beta testing can be rewarding and enjoyable. It is a great way to extend the boundaries of your hobby. You will meet some very knowledgeable users and maybe learn something new. You will need to give up some of your free time but that shouldn’t be an issue if you are doing something you enjoy.

Unfortunately, you will also need to develop a thick skin. Users will constantly tell you that you don’t care about their bug. That you are a hack and have no business as a tester. Just remind yourself that you set aside a little time to help grow and improve your hobby.

And you did it to help improve the enjoyment for that user that just called you useless. :wink:

23 Likes

I agree with you 100%. When I first read that MSAsobo was offering a chance to take part in this beta testing, my biggest fear was that this would happen in sequence with SOME (not all) testers:

  1. Beta tester reports problem that has been going on since day one (several choices here - Live Weather, ATC, plane performance, graphics, etc).
  2. MSAsobo doesn’t address or fix their specific issue that may or may not have been on their list to address for the entire beta testing period (or God forbid, makes it worse with the next release).
  3. User reacts: “Well I guess MSAsobo doesn’t care about what their beta testers report. I’ll never waste my time with THAT again.”

Since I am a Steam user, I’m not eligible now but I admit I’d be a terrible beta tester anyway. My flying needs are simple, and my understanding of the more complex principles of flight are limited, so I would be of no help. I really do hope that everyone ends up happy that they participated in the process, but my guess is that there’s no way that’ll happen.

2 Likes

A summary…

If reading for about two minutes is too much, don’t join a beta flight.

12 Likes

Is the beta open to Steam users? I only see an option for MS store users of MSFS.

I believe I answered that.

I agree ! and… the folks that actually effect change are not looking at most of these “forum” discussions anyway. They are most interested in what is being reported in ZenDesk and internal testing…

3 Likes

Steam users should now have access to this public flight test. @magnetite2

https://forums.flightsimulator.com/t/pre-release-testing-su7-update-public-flight-now-also-for-steam-users/480068

8 Likes

This is good news.

3 Likes

Just in case you change your mind.

Well, I wouldn’t say that is true. I may not have in depth knowledge of the complex systems, but one thing I experienced was sudden transitions, was able to reproduce them, so I can help by testing that in the beta. I can also test other conditions that may cause transitions. I can test the other bug fixes but in terms of where I fly usually, because those places may have certain characteristics, or perhaps XBox may treat things differently than PC.

The important thing is to test the list of things that were addressed, and state objectively whether they passed or failed. Personal opinions on further changes are feedback that can be looked at for a further release, but the focus is does it do what it was intended to do.

And then there is everyday usage, is something entirely new bug generated that would be considered a show stopper, that would be deterimental.

This isn’t the close to all inclusive bug fix release that SU8 is intended to be. This is just testing for something that is solid enough and livable for the holidays. So the things I listed above, they don’t require an expert. It just requires someone to be objective and focus on testing a list of fixes, and looking for any potential show stoppers.

2 Likes

Absolutely agree I am passionate, as much as anyone, and want this Sim to succeed, however I do not think I am qualified to be part of the beta testing.

I am not a meteorologist expert…yes I can fly (simming) and see clouds however I would never comment on how realistic it is, and at the end of the day they may look pretty, but is it realistic? We all want the best for the users…we can agree on that.

I just Love simming and learning something new every day and want this to succeed, and to be realistic as possible.

The only experience as a Pilot is in a Cessna 152 ten lessons, or as a passenger on GA and obviously airlines…again this does not qualify me as a beta tester.

However as a Simmer I am obsessed in aviation, in my Lunch hour at work always reading books on aviation, but does not qualify me.

Think actually pilots have more acronyms than the medical profession.

2 Likes

Anyone who spends their life experiencing the weather in one form or another is qualified to provide feedback to the non-expert developers of a $60 video game. Some good points made here and we appreciate the expertise folks share, but Microsoft/Asobo is soliciting public feedback. It’s their job to filter the noise and bad info for any value, not the user’s responsibility to conform to your beta tester ideals.

10 Likes

That’s quite a slippery slope viewpoint to take. What’s next? Are we going to say that anyone who has never actually flown a plane is not qualified to determine if a plane in a flight simulator works properly or not?

Although you make some good points, Asobo/Microsoft are soliciting public feedback. Let’s try not to gatekeep who is and isn’t qualified to beta test. If you choose to beta test, report your feedback, and let Asobo/Microsoft filter through it.

11 Likes

How does a Steam user get a Beta access code to enter on Steam Beta menu so we can load the Beta???

I was confused by that initially, too, but no code is needed. Once you select the beta in the drop down box, steam downloads a small update. Wait for it to complete. Then launch the sim from steam and MSFS will have the full update. Mine was around 715MB.

There is one benefit of an open beta like this which shouldn’t be overlooked which is that with no NDA restriction the forum and the user base itself become the filter.

Yes, there may be a target list but if a group of experienced beta testers focus only on this then it becomes easier to miss things, especially those which may not seem directly related to the hit list.

With a broader base of testers it’s possible for even a lesser experienced tester to notice something and then ask on the forum if anybody else is seeing this.
Which may prompt some with a little more experience to go back into the beta and verify.

The key to this is communication to be open and experienced testers to be listening.

3 Likes

Good point well made.

1 Like

@willisxdc Thanks for making this first post.

I, too, find beta testing, on the whole, an enjoyable experience. However, it does require a different mindset.
I also agree with the thoughts expressed in this thread that a broad base of beta testers benefit the test as more issues are likely to get identified.

I totally agree with the point made that any feedback should be as clear and relevant as possible, ideally with enough information for someone else to go and try to reproduce the same issue that is being reported.

On a more general note, it would be of great use, even on the general forums, if any post made concerning an issue was as comprehensive as possible in the information provided. Such posts do require effort and thought. Such posts are massively more useful in trying to identify root causes (or at least the circumstances needed to reproduce the issue) and consequentially increase the likelihood of fixes being found.

Concise, clear feedback benefits everyone.

5 Likes

Guilty. I could have worded that better. My intention was to illustrate that we all have our areas of expertise. That is different from areas of experience. We have all looked up at the sky and have a pretty solid understanding of what clouds look like, from that perspective. That is experience. Fewer of us have viewed those myriad formations from above. Even fewer have seen some of the stranger formations that occur far more frequently than most would expect.

I am a proponent of the open beta testing. The more eyes on the sim the better. Many users have spotted issues that some of the most experienced pilots among us have missed. You may be a professional UI/UX designer but didn’t notice something as simple as a reversed switch in a menu.

I only ask that when reporting in a test, or in the general forum, that we only assert “reality” when it is within our area of expertise. As I mentioned with aircraft testing. If you have never flown the real thing, you cannot determine if it “flies right”. You can test the aircraft against the numbers, however. Both methods are valuable, and welcome.

If you are testing a beta and see something that doesn’t “look/feel right” then you are testing within your area of “experience”. In this case the correct report would be, “This is what I did, this is how you can reproduce it, it seems strange to me, is this right?”, as opposed to testing within your area of expertise, in which case your report would read, “This is what I did, this is how you can reproduce it, this is incorrect, it should be like this.”

@SkipTalbot
I do not want to “gatekeep” the process. I welcome all that want to step up. I simply want to make users aware that testing is, as @tamalien said, “a different mindset”. While the previous suggestions may be an “ideal”, I do not expect everyone to “conform”. I only caution some that would join a test flight, that it may not be what you expect. Many have expressed their discontent when the developer ignored their input. While we as public testers may not be focused on specifics, the developer is. To get the most from your participation, it is best to take on that same mindset.

In summary…
All that wish to participate are welcome.
If you have a relevant area of expertise, please, assert that “expertise”. We all benefit.
If you encounter something relevant to your area of “experience”, please, report it. We all benefit.

4 Likes

There’s no way you type from your phone…:grinning:

1 Like