As a 3rd Party Developer, I would propose the following to Asobo/Microsoft to ensure future updates don’t go overlooked:
New aircraft additions like the SR22, A320, Etc are not publicly available to test in Beta Flighting for at least half the beta. My recommendation would be to hold a separate beta.
I fear the instability of SU-14, especially in consideration to navigation, electrical systems, and multiplayer issues may have been alleviated if the beta testers were brought in to test stability of the simulator solely and not the additional aircraft. More testing on the core system changes needed to happen prior to SU-14’s release.
I know this will be an unpopular opinion, so feel free to disagree and keep things civil. I just had to put this information out somewhere in case anybody part of that internal process reads this. SU-14 is rising in the update leaderboards for it’s bugs, CTDs, and maintenance cycles required for 3rd Party Developers.
Going to be a long month for our team in order to get our add-ons back up to par. Not to mention that this update is Asobo’s “Holiday Update”. So our aircraft will be riding these changes for an unforseen amount of time. Stability is all I wanted for Christmas. Instead I got Krampus.
Do you have a feel for how many third party devs download the public betas and test their products prior to launch? I’ve seen some do this and have patches ready on release day, but I 've also seen some devs that wait until the formal release and then test, I assume so they don’t “waste time” on evaluating a beta release that could change again.
We don’t waste time. Usually, we are developing new addons that aren’t public yet. So installing a Beta Version may hinder new development as it’s notorious for resetting settings, forcing re-installs, and breaking dev-mode features.
We wait for public release when things are guaranteed to have changed and stayed changed.
With the amount of people in the testing program, I don’t think it would have made a difference. I don’t think the amount of bug reports would have been different when it comes to the core sim and that the bugs that needed to be reported have been reported.
In this case, issues with the aircraft were sent to iniBuilds and other sim-related issues were handled by MSobo. I don’t think one took resources from the other.
The release notes are specifically provided at the beginning of the Beta, and updated with each incremental build released during the Flighting.
The beta collects feedback from testing of those note points by the public testers.
Third parties are not required to participate, but are certainly encouraged to do so. Their approach is entirely independent with regards to beta changes specific to their products.
On my end, I have both the Store and Steam versions installed on the same system. One is on Beta and the other is on Production. This enables us to test things in the new environment while keeping my main sim (Store version) on Production.
Also to mention for those who don’t know… In order to send packages for Marketplace ingestion, the package needs to be build using the Production version of the sim. This can make it hard when developing using the Beta version only.
This bites anyone with a big catalog. At a certain point, there isn’t much time left for new developments and all resources are spent on fixing things that got broken or adding features the community expects us to support (e.g. CFD)
Are you asking for a separate beta program for updates to add-on aircraft from the beta for updates to the base sim, on the basis that this would make it easier for you as a third-party developer to pay attention to the changes in the base sim without being distracted by other peoples’ aircraft?
(I’m simply uncertain what opinion it is that you have; can’t tell if it’s unpopular or not. :D)
Some Third Party developers will test in Beta using Community Folder versions of their content (i.e., the folks who exclusively sell through Marketplace). That’s one way to see impacts to the portfolio.
I completely understand this point. Resources are finite, and I do not envy the developers who try to do this full time and regularly have the platform on which they depend change something that breaks their work.
I would put out there that maybe this is something to try out next time even if it’s not your typical process. Try out your products with the Beta (risking some wasted effort), and get an early look if there are any game-breaking priority 1 issues. Then at least you could prioritize. Perhaps there’s an issue that does dictate you need to pause current development to prioritize for a fix on an existing product?
Again, this is all very easy for me to say, I’m a consumer, not a dev. As a consumer, I’ve understood that after a major update, I’m going to need to wait for updates on products I have. I don’t get too bent out of shape over it.
We strive to read every post on the forums, and I can confirm we have seen this topic. I have passed on a link to this thread and your suggestion about future betas to the Production team for their review and consideration, but to set expectations appropriately, this is not a commitment that we will change the way betas are conducted going forward.
Regarding your point about SU14 stability, this is something we take very seriously. Obviously individual results may vary and some players (and/or 3rd Party Developers) may experience a disproportionate number of issues, but we review internal telemetry data collected from thousands of community testers before each release to ensure stability is on par or better than the previous release.
Please let me know if I can offer any further assistance.
After today’s preliminary findings, here is our biggest concerns:
Some Lights don’t have working indexes in Multiplayer including index 1. This is causing major issues for light systems and mutiplayer systems that took advantage of these extra variables.
Loading Tips must be under 200 characters or it will cause CTDs
Leaving Max Mach at 0.00 (flight_model.cfg) will result in uncontrollable controller vibrations during flight.
Airbrake Scaler has been overhauled. Any aircraft with airbrakes/spoilers need to be re-tuned.
Sound Bug: Some of our aircraft have idle engine sounds while engine is turned off. Unable to confirm if Engine Combustion/RPM is being properly transmitted over the multiplayer server.
Wanted to add my findings after this SU14 / Bell407 / helicopter flight model discrepancies not playing nice with CowanSims models now. Haven’t tried out my GF mini 500 yet, but hoping it will be ok with its own FM.
I get the request. Hold off on beta’ing any planes that are about to be released so that beta users test the third party developers planes while they continue development.
And that does work out for third party plane developers. But then the actual plane itself doesn’t get tested fully. Because keep in mind, if there is a problem with the plane that is to be released in a SU, no fixes will happen until the next SU, which is 3 months or more. Third parties can make releases whenever they want to. A fix to the sim HAS to occur with a Sim Update. The Community and thrid party developers asked Microsoft and Asobo to not do hot fixes anymore.
I think a more practical approach would be that third party developers should have community volunteers. Before a plane is released, third parties already have trusted testers and YouTubers that they give the pre-release planes to in order to test and preview on YouTube. These people aren’t developers. They can install the beta for the SU and test the planes.
I am also a developer. So I get needing to have a stable environment to develop against. I get why third party developers have issues testing SU’s internally. But I don’t think shifting things so that now planes that are about to be released can’t be fully beta’ed is the answer. I think it comes down to having definitive volunteers in the community who will test against a beta.
As a consumer, I’ve understood that after a major update, I’m going to need to wait for updates on products I have. I don’t get too bent out of shape over it.