Snoop dog joined the code team for the 787 in SU4. It’s not a bug it’s snoop being snoop. For shizzle my sizzle
1 Like
Yes, I guess if there was not any incentive, than people would not do it.
Still, if you think about it how much money that saves Asobo/MS on paying testers. Guess their suits will receive big bonuses for this.
I would propose to take it a step further:
‘Microsoft Flight Simulator Coding Challenge’
Description: Members of the community with coding experience are invited to apply to contribute to our sim core code. We will hand-select the lucky winners from the pool of applicants, most likely just people with decades of coding experience. The lucky winners will help us to code the sim. They will do it for free and receive nothing, they will just be ‘lucky’ to help us and bring our fine product into maturity state.
It’s a good thing, but was just wondering if in the beta proces the testers are allowed to use anny 3rd party content (scenery, aircraft, aircraft mods etc).
It seems that nobody looked at LFPG airport.
I’m a little surprised that the “Offline AI Slider” bug in the menu was not noted, as a beta tester maybe you should also consider checking that all menus are free from bugs ,otherwise they have done a good job.
1 Like
Yeah, but the entire game industry uses voluntary beta testers. The software industry uses voluntary beta testers too (ie. Microsoft always releases an early version of Windows to beta testers). Rather than pointing at Asobo as the culprit, maybe you should go after the entire gaming industry, and the entire software industry? Asobo is just doing what every other game company and software company out there does.
It’s not just the beta testing that was a success, but also Asobo actually listened to them and didn’t release an Update with glaringly obvious bugs as we’ve seen in the past (well the UI one is but let’s gloss over that!).
It’s further proof that had Asobo had this beta team - or preferably an opt-in branch - in place since day 1 then we’d be so much better off than we are, hindsight is great but many of us were saying a beta team was needed as soon as that first update dropped with its multitude of issues…9 months since release, 9 months of lost opportunity to release better updates.
But credit where credit is due, Asobo have clearly learned from their mistakes and this Update at least proves they are heading in the right direction with regards to this area. Let’s hope it continues.
3 Likes
5 Likes
To understand how well the work of the beta testers was carried out, we need to see their reports for the entire testing period. What bugs were reported, which ones were fixed, which ones were missed. Otherwise, the conversation is nothing
3 Likes
Not going to happen. They always keep the beta testing under wraps.
But with all the changes Asobo put in this last patch, and the fact that there were no game breaking bugs like previous patches with so many changes, you can infer that beta testing was successsful.
1 Like
Well this is an idea, just mark the release as beta.
And just give us the ability to roll back to the last (stable version)
Just like Xplane 11 did
Then users are not forced into the latest version of bugs, let the users decide if its worth it ?
I am off to have another CDT…
2 Likes
I just want to say that so far, it appears SU4 was a very successful patch.
There are few major game breaking bugs reported for it from what I have read so far.
Which one is it?
yes sir! It’s been a long while since I’ve seen that movie
1 Like
Fo shizzle sho.
1 Like
So here’s the thing… Microsoft has a QA team for Microsoft Flight Simulator listed in the credits. The credits are out of date, so I don’t know if that team is still involved, but even if they aren’t there will be others.
These devs likely look through bug reports, and serious known issues to try and find the root cause which they either fix or pass on to the relevant developer/development team.
However, this team will be small(ish), and for reasons, I will cover below, using volunteer testers is probably better than having a large QA team, and it is for that reason that most software companies use external (often volunteer) teams.
It’s easy to test software that will run on a console. While there may be a bit of variation today on the hardware and software stack, it is fairly limited. Home computers on the other hand is an utter crap show. There are hundreds of thousands, if not millions of possible combinations when you start considering things like:
- [Hardware] Component combinations
- Age of hardware
- Health of hardware
- BIOS versions
- Firmware versions
- Hardware configuration (settings)
- Software versions
- Software configuration (settings)
- External software or peripherals that probably shouldn’t but still often conflict with your software
- The game’s settings
For games that require a network connection such as MSFS, you also have combinations in:
- LAN Hardware
- LAN Software (firmware, device OS)
- LAN Configuration
- LAN Settings (On the switch, router, access point, modem etc.)
- The USERS network interface (ie. WIFI adapter hardware, software, settings etc. || Ethernet connection, the cable that is used to link into the network etc)
Plus then the:
The WAN (Wide area network) including possible issues like:
- Your ISP, the speed they offer, do they throttle your connection, do they drop packets regularly
- Optimal routing routes
- Network congestion, especially if you live in a place far away from the servers and/or that only has a few outside connections to the outside world (think people who live on islands)
That is a lot of possible combinations. Not all will apply to every software package, but here is the crazy thing, I came up with this list off the top of my head, so I am quite possibly missing things. A console eliminates lots of the hardware-specific (first group) combinations.
But when you launch a PC game, especially a complicated one, a small number of internal testers will not be able to provide enough useful information. That is why software developers often open up their software in one way or another to volunteer testers. The company doesn’t have to pay for or worry about:
- Gaining enough testers from their target geographical markets
- Provide an extremely large array of hardware combinations (ranging from below min spec to way above)
- In the case of MSFS they don’t need to find a QA team/org that has a wide range of aviation experience on top of their QA backgrounds. That certainly helps of cause, but you won’t find tons of people who fill that niche, so using public testers can help with the aviation side, where QA can focus more on the code side
In return, your innovators and early adopters get access to the product early, and they get to shape the product through their experience and feedback, something early adopters typically enjoy doing and something which benefits everyone.
I suppose the developer could pay these early adopters, but there is no reason to because given the chance they will happily have an unfinished product, so long as they can get it early and help shape it, and not paying them keeps costs of RND down and therefore allows the product to be sold at a cheaper cost…
There is one other issue I want to touch on with internal testers. They often have the same or very similar computer steps removing the randomness needed when testing computer software, and they often have very strict testing guidelines and goals (they don’t tend to mess around as much).
There is a very hilarious example where that can go horribly wrong: The Eve Online Trinty Expansion (2007), where a bug that wasn’t caught by QA in a new patching program caused the system to delete the boot.ini files on users computers thereby stopping people from being able to start their PCs.
CCP Games launched a thorough investigation and released a dev blog (the linked above) detailing the cause, and more importantly why the bug wasn’t caused. Here is the except with the key part bold and underlined:
Why didn’t you catch this during testing? It’s partly the reason above, not enough time to test the graphics content upgrade thoroughly to notice it removed this file. We also discovered that we didn’t have enough variation in our hardware and operating system setups since Windows will recover if it’s on the first partition of the boot drive. It seems that most computers at CCP are set up this way and this was my personal experience in the evening of the release. I upgraded my Revelations 2.3 client to Trinity Classic and from there to Trinity Premium. I logged onto Tranquility, then logged directly out again and rebooted my computer without any visible problems. Needless to say, we have already revised our testing procedures to make sure this does not happen again.
You can view the full article here: Eve Online: About the Boot.ini Issues, and a hilarious internal video made by the CEO and released to the community on the company vacation: https://www.youtube.com/watch?v=msXRFJ2ar_E.
Anyway, needless to say, public testers are good and useful. If you don’t like the process you don’t have to apply. But I hope this rushed post explains the benefit of public testers.
6 Likes
Back that statement up with specific features and quotes from Asobo saying they would be in the update, aside from the Working Title integration that Jorg and Will apologized for. (They found significant hurdles that need to be resolved as well as a number of upcoming features that Will wanted included.)
Given the number of testers and the number of airports, aircraft, systems, UI, etc, I would suggest it to unreasonable that every airport, aircraft, avionics, flight model, UI setting combination could possibly be tested. Some beta testers put more than 80 hours into testing what they could and most tried to stay within their area of expertise.
Anything else you would like to point out that they missed?
As a Trusted Forum 3rd Party Dev I am surprised that you were not part of the testing. Maybe next time you can opt-in and have the opportunity to not only read the reports but also test your own software in the beta so you don’t have any embarrassing bugs turn up.
I am surprised at the amount of disrespect being leveled at the people that volunteered their time to try to help improve their hobby. Seems to be plenty of users that have lots of time to moan about how bad everything is but they don’t seem to be interested in trying to do anything to improve the software.
There are lots of forum members that ARE here to try to make things better for everyone. They ‘volunteer’ their time in the forum trying to help other users get past some of the system hurdles that always crop up. Others that dedicate their free time to help new pilots learn their way around the aircraft. Many are spending as much time here helping the less fortunate as they do enjoying the sim.
Occasionally the opportunity comes along to share some hard won knowledge and expertise that can benefit an entire community. It is unfortunate that there are always those that are so bitter they cannot just say thank-you when some try to help. They need to continue to tear at the fabric that makes a community work.
I give a nod to @gadwin777 for stepping up and acknowledging those users and share the sentiment. Well done Asobo for flighting this update and well done Gadwin for saying thanks.
5 Likes
I’ve been on many beta testing teams and the bottom line not everything the beta team identifies gets resolved in time for the release for so many reasons.
1 Like
A quick point of clarification there is a difference between “Trusted Third-Party developers” and official partners.
Official Partners apply through the Microsoft Flight Simulator website, and while I am not privy to all of the benefits of this program, the main thing is these people can sell their products through the in sim marketplace.
Trusted Third-Party Developers is a system that was proposed by, developed and now administered by the forum moderation team. The role allows developers to publicise their products in the Product Announcements channel, and through a custom integration, it also posts their announcements to the official MSFS Discord server.
The Trusted Forum 3rd Party Dev role (unless something has changed since I left the moderation team) doesn’t provide any additional access to the developers, or early release builds of the simulator. That is not to say that a Trusted 3rd Party Dev can’t apply for and get access to pre-release builds, but it probably isn’t guaranteed. Anyone who does apply for the Trusted Third-Party Dev role on the forums should definitely consider applying for the official partnership program as well (first link).
In the meantime, I would definitely like to see some representatives of each official partner getting access to pre-release tests as well so they can check their products and ensure they can submit patches to the marketplace team before the patch drops. Whether or not that happens, I of cause can’t say because of an NDA, but if you are an official partner definitely consider reaching out to your MS contacts about the next test period.
3 Likes