Forum Community Question - How does internal testing work at Asobo? Does Asobo have dedicated full-time Q&A personnel? What are the requirements for a build to pass testing? - Timestamp
Martial - We do have testing. If I had to describe the test process, it would be like a rocket with several stages. The first one is at Asobo: We are here to create the build and to test that. Then, Microsoft test teams can have their hands on it. Then the last stage of the rocket would be the certification stage at the end of the process to ensure that the builds are compliant with the requirements for Xbox, for instance.
Martial - So what we are doing at Asobo: The team is split into several parts. I will start with the first part, which is the build and release management team. They are creating packages, they are creating the executable and make sure it is working on every platform: Windows Store, Steam, and Xbox. So this is a team of five.
Martial - After that, we have a team of 15 people to test the content. So the very first content is the flow, itself, to test the features: Everything including UI/UX [user interface/user experience], sales, [menu] selections, everything like that.
Martial - We have a team of five plane specialists, and three of them are actual pilots. Most of the time, we are trying to get pilots who are in between situations, like Airbus pilots that are waiting for their missions, for instance. We added a team commercial pilots. It’s not the case anymore. But it should be something that will happen very soon. So we’ve got five people on the plane team.
Martial - We also have three people on the world test team, with one coordinator. These guys are taking all of the assets you’ve seen presented by Jorg and are testing the implementation in the sim to see if it’s fitting well with the landscape and if everything is working, if streaming is working.
Martial - After that, we’ve got a small team testing the airports. So same here: Everything that has been created by partners to make sure they’re included in the sim. To make sure we have the right navigation data. After that, I can talk about the input and VR [virtual reality]: It’s pretty hard to ensure we are compliant with all the inputs and outputs we can find on the market. We have several headsets, several HOTAS [hands-on throttle and stick], and yokes. These are distributed to the whole team. We have some coordinators that make sure everything has been tested. We can’t guarantee that it’s the case for all assets because there’s so much hardware in the market.
Martial - I will conclude by talking about the people who are reading the large community, that are collecting all the feedback. We are in a super important part, here, because it is almost impossible to test all the assets. To test the whole world every time. To test all the hardware that is available on the market. So that’s why we need the community to be helping us. That’s why we introduced the flighting sessions.
Martial - Two testers that are monitoring the community activity to ensure that everything is going fine. And also, I’d love to talk about three other people who are collecting feedback in the tools we are using, which is Zendesk. If you do the math, it is more than 20 people who are working on quality assurance. (I hope I am not forgetting someone!)
Jorg - On the Microsoft side, there are 50 folks testing in various ways or interacting with bugs. What’s fascinating about the product is, we have 10 people by and large, sometimes more, dedicated to the Marketplace to make sure that things work inside the Marketplace. We test the planes that they actually they fly. The third parties have been so impressively creative and their output is so huge that it’s starting to – I wouldn’t say “worry me” – but just: Recognize that there are already 1000 addons. Recognize that there are about 100 things coming out every week. Our team is growing. The development team is getting larger and larger as you probably know. We are 250 or something. With that, there is more and more surface area. In my mind, we need your help. There is no way we can learn everything about everything. Every single plane in the world, every airport. So we will really start to be more structured about getting flighting feedback in specific areas. My dream would be that some of you help out with specific planes before they come out. Some of you help with airports or whatnot. And that would be a great collaboration. We have some tools coming, as you know. There’s the airport scenery tool so you can fix your own bug. If you find something we don’t know about, just go fix it and submit it to us. So the surface area of the flight sim is just huge. And I think we will continue to grow and add more people as we need to, but it would be terrific if you also help us.
Martial - I forgot two members of the team who are dedicated to dev mode and the SDK.