Testing within the Microsoft network is an ok place to start testing, but does NOT constitute testing. Testing must include testing as users would see it, and Asobo/Microsoft MUST build and create test plans on PC’s and Xboxes on a non-Microsoft network before they can consider their testing regimen complete.
This could be a major issue in their test plan if they’re not already doing it. I have a hard time believing they’re not already doing this?
easy to answer most simmers don’t know how to fly “near study level aircraft” so a lower rating. The CRJ was 3.3… until the dev said to read the manual and watch some videos on how to fly. Right after the rating jumped to 4.3. The difficulty is high on these aircraft. I would not base a buying decision on the rating. I would base it on if I had the cranial capacity to understand it.
They may be doing it for the releases of the sim, they aren’t doing it for store releases. They test on their test servers, the package it with encryption for the i game store. The thing is, these two addons are probably calling some API function that not everyone is, or calling it differently. You would have had to have a client be outside the Microsoft network, and calling the API function exactly as these packages do, in order to find that bug. It’s a missed test case. It happens. Comprehensive testing is only as good as the test cases you have. But you can’t paint is as a failure of the entire testing methodology. It was a test scenario no one thought of.
It is disappointing we have to wait, but I a, not going to fault a single party in this at all. This was an obscure bug that was pretty easy to miss.
I’m not sure I understand your response here. I totally get what you’re saying about API’s being availble internally, not externally. I got that the first time.
All I said was, IF they don’t have any test boxes external to their network, they’re not actually doing any real testing, and they need to address this. Testing on their internal server network would be the first stage. But, you can’t learn what problems users will have until you test outside the development environment.
This is not an obscure bug. It’s a major issue with their test regimen if they don’t have test machines on external networks. And it would be ridiculous if they think that testing in the development environment is enough.
The process of testing each addon before they package it to place into the marketplace is already lengthy enough. It takes about 5 hours for each package per platform. So if an addon or update is meant for both pc and Xbox, that is 10 hours. Adding in additional testing of testing on platforms outside their network would add to that 5 hours. People already complain that the marketplace, it takes too long to get updates out. Out of 600 addons so far, this has happened to two products (Milviz is a different problem that Microsoft can DRM code the Corsair for some reason for Xbox). So that is 0.5% of the products that have had this issue, to add a couple hours more testing per product and slow down the whole update process , is that something you really want? Sure, they could have found these two products early before release. But any addon you are waiting for an update for, you are going to experience more of a delay with this new testing per package.
I’ll ask you this. How many hours have we had to wait for a fix for these two addons that had the issue, two VERY important addons that lots of people are waiting for, that would have been caught if they skipped the test on the development environment and tested solely on a computer and Xbox that was connected like user’s machines would be?
How many users had to deal with, “Hey, why isn’t this working. What the heck is wrong with these products?”
What happened to Asobo’s and the developer’s in question credibility when these bugs slipped through?
Was this problem worth saving those 10 hours of testing, during which they skipped a huge portion of the test process? Why bother testing at all? How many sales have been lost due to the issue? Would those losses and loss of credibility have paid for that testing (granted, across all products)?
There’s no excuses here. Their test process is flawed in a major way. Skip the test in the development environment then if you’re worried about those 10 hours.
Your statistic on the % of products with problems is also flawed, and an example that you can make statistics say whatever you want them to say.
I’m sorry, I cannot accept the excuses for why these issues weren’t caught, if they would have been caught had the test regimen been properly specified, and, if, in fact, they don’t do the same testing outside of their environment.
It would not be as simple as simply “skipping” the inside network test. Those Xboxes would have to be inside the Xbox network to load the addons initially. There is NO way to load an Xbox with an addon into the reserved space for the app from outside the Xbox network. Otherwise developers could test addons themselves. The security of the Xbox architecture won’t allow that.
The presumption is an Xbox should act the same inside the network as outside the network since they are the same hardware and same software. This turned up a problem in the API or core code. Once that code is fixed, an Xbox inside or outside the network should act the same.
I understand people don’t like buying a product then having to wait. But on the other hand, if the DC6 worked out the gate on XBox, but next month a sim update required PMDG to make a change, you would want that update as soon as possible. If instead of 2 weeks to get the update, it took 3 or 4 weeks, you would be mad and want them to optimize the process to get updates into the marketplace. The are mutually exclusive. You either get more testing, or faster packaging time, but you can’t possibly add more time to each package but make the updates faster.
In case you think I have no skin in the game her, I own both products. I haven’t asked for a refund, even though Seafront Simulations was worried enough to ask all of us to do so. I understand the complexity of what happened, and I’m not blaming anyone, not the developers, not Asobo, not Microsoft. It will get fixed, and I will be able to enjoy my products. Meanwhile, there is still plenty for me to do in MSFS. It’s not like the sim is crashing because of these two products. Look at it like these are products you really believe in, and have pre purchased them.
I’m thinking a few hours more testing beat the almost total fail of the DC-6.
dw
I would of not had any problem with a pre purchase. I’m sure I would of put my money down for the DC-6. I enjoy older aircraft. The issue in my mind was putting a product in the market claiming it worked as advertised.
I spend many many years as a test engineer for a very large company. The bottom line is you test a product in the environment it was meant to be used - full stop. There is no other test more important.
dw
1] They are testing on an XBox, just one that happens to be inside the Microsoft network
2) We don’t even know if it is physically possible to load and then run outside the network. Maybe the test xboxes have their “drive” on the network
3) Once this issue is resolved that makes these two packages behave different, the next big product everyone wants, or the next fix the DC6 requires because of a sim update, everyone will be demanding Microsoft do what they can to speed up the process.
You as a test engineer, is every single system you test guaranteed to be 100% free of bugs? Of course not. No test engineer is going to stake their job on saying “there are absolutely no bugs”. You aim for the best you can. 99% certainty. Well guess what, 600 products, 2 products have had this bug happen, that’s 99.5% of the products this process has worked for. So now we have found a bug in a core resource. Once that is fixed, these two products will work fine.
If this had happened to the AI shipping and let’s say another Carenado plane, mo one would be making a fuss about it. But the fact that it happened on the plane everyone wants, well now all the experts want an improved testing process. But no one even knows if that is even possible. And some point in the future, what everyone is asking for, when it become inconvenient to them because the newest favorite product has a bug they need an update for and they have to wait, then everyone will be calling for changes again.
All the parties involved, they are actively working in resolving the issue. Night lighting, one month a new nightingale comes out, everyone criticizes it, the next month a different night lighting based on feedback comes out that everyone loves. Everyone is complaining about live weather now, next sim update I’m positive changes will come out that people will like better. Everyone involved is very responsive. The DC6, it will get resolved.
Just for the record: I just purchased the DC-6 for PC (well, iMac… but that’s another story ;)) and took it for two test flights, one from Zurich to Geneva (LSZH - LSGG) and another one from Anchorage Intl to… uh… Delta Junction (D66)… when I quickly realised that my noob landing skills were most likely not up to the task to landing a DC-6 on such a short runway (2800 ft).
So I diverted to the nearby military base Allen Army Airfield (PABI) where I still put the DC-6 into the snow, just before the runway 
But it was a blast, and other than the crash landing I figured out the basics, I think (with the great help of Mr. Copilot, of course).
The only thing that I missed from the start was the interactive checklist which would guide you through the required steps to start/stop the engines. You know, with indicators of which button / lever to press (the checklist is currently completely empty).
I do realise that you can either use Mr. Copilot (via the tablet) and have him/her set the airplane into the desired state. Or I could read the manual (which I eventually will) and/or watch tutorial videos (which I also will).
But one reason I came back to flight simulators was this “guided checklist for dummies like me”-feature in FS 2020 (the interactive checklist where the knobs and switches are highlighted for you etc.) and it would be nice if an upcoming update of the DC-6 would support that FS 2020 checklist feature. You know, as some “middle ground” between “full Mr. Copilot automation” and RTFM
In the meantime, here are some first impressions of mine - enjoy!
Take-off in Anchorage (PANC):
En route to Delta Junction (D66):
P.S. So far neither did I encounter any crash to desktops (CTD) nor any unusual framerate drops (but then again, I am not the guy who is measuring FPS with a counter either - so FWIW). All control elements worked as I expected (but then again, I did not dive deep enough to make any qualified judgement on that - so take that for what it is).
Guys, I got the DC6 from their website when it releases and recently it started making CTD. Not sure since when beczuse I haven’t launch the sim for a few weeks.
Any fix that solves everything xD? I can’t find answer online.
Disable DX12 if it’s enabled…
You don’t understand what a Test Engineer job is. As a Test Engineer you write tests to test a product as it is to be used in the manner the company states it doses. For instance my company states a new braking system on this car will stop the car is 6 feet. My job as a TE is to write up a test to put the car in the real world the car will be used it to see if it will stop in 6 feet. If the TE uncovers problems it is up to the company to correct them and then re-test till the car is stopping in 6 ft.
MS did not test the DC-6 in a real world test. They, both companies, assumed. I smile at the folks the justify this issue as being ok. My complain is not the product but the process. We users put money down on the belief we’d get a WORKING plane. We did not. No amount of pointing fingers changes that fact.
dw
And you are ignoring the fact that it may very well not be possible to test outside the Microsoft network. You know there is no way to access the reserved space of an Xbox application. That is security so that games files and game saves cannot be tampered with, making it a competitive environment. Developers themselves cannot test their addons on Xbox, they have to run on PC in DevMode and use the “simulate Xbox” menu option to test.
So, in the Xbox test labs, there are Xboxes that they can load into the reserved space. But is that because the hardware is wired into a network drive instead of a hard drive. If that is the case, the “hard drive” of the Xbox can’t be moved out of the Microsoft network, since it IS a network drive.
We don’t know. I do know they will work out the issue. I hate to say it, but if you want an environment where not only the developers but Asobo and Microsoft can easily test addons, then you probably have to go PC. I’m not saying that to be rude. But XBox is so locked down to ensure competitiveness between players, it makes some of what you insist good engineers do impossible. If you can’t live with that acceptance of risk that some Xbox products may have delays, then pc really is your only option
I never take any comment personally unless you say this is personal .
OK let me say if I was a TE for MS I’d suggest putting an xbox in someones home attached as a ‘normal’ user would be. After all the internal testing do the last test at the ‘home’ xbox. Seems very simple. The xbox would be in the ‘real’ normal environment. The one users use.
As an aside I’m generally very happy with the game. I am amazed they MS has done a great job considering the task. just not happy to be out $50. I do want and love older planes. I’m an dude that has aged out of real life piloting.
dw
You are my hero! Works
The problem is, it isn’t simple. You have to inject each addon into the reserved space for MSFS. You can’t do that within the Xbox itself. That is why I have to Imagine this “Xbox” they use for testing doesn’t have its own hard drive, it probably uses a network mount to simulate a hard drive. That way from some PC, they can inject into the reserved space of the test “Xbox”. In other words, you can’t just pic that Xbox up and put it somewhere outside the network, it has no physical hard drive. You can take the Xbox out the lab, but there will be nothing to boot, it would just be a shell.
A lot has gone into making the platform as secure as possible so people don’t modify save games or add equipment or mod the game files themselves to give themselves the edge in competition. It’s terrible that people would do that if they could, but it would happen. Esports would mean nothing then. So the Xbox architecture is locked down really really tight. But people keep forgetting how secure it is, that this testing should be really easy. And it’s not. If it was easy, someone would have figured out by now how to load addons onto Xbox themselves, and it would be all over YouTube.
I’ll never understand this. Microsoft, the owner and developer of the Xbox platform, OS, console and not to mention the simulator (w/ Asobo), can’t figure out a way to test add ons securely with live Xbox consoles in the “wild”? I also purchased the DC-6 last week and it took exactly 10 minutes to discover it didn’t work. Wouldn’t have taken much time at all had they found a way. That is an epic flaw in testing procedures however you want to look at it.
