Forum Bug Report Management

Discussion Hub General Discussion
,
1

This is a long post, but I’ve put care into what I’ve written here. It’s not a rant.

I’ve been a member of the forum since Fall of 2021, shortly after the original Xbox release of MSFS. Since that time, I’ve invested heavily in the process of reporting bugs and beta testing. As I’ve mentioned a number of times, I held a position in QA for a now defunct software company that developed flight simulators and other video games. I moved on to producer positions within that same company and other companies as I furthered my software development career. Because I got my start in QA, I value what that oft-maligned department does for the development process of a software title.

I have spent a considerable amount of time entering bugs on this forum – both for the “Live” release and beta releases. The process of entering a bug isn’t a quick one. Often, when I see something I deem to possibly be a bug, I have to reproduce it first. This involves possibly reloading a flight or a C&D state or even reloading the sim. Then, once reproduced, I often have to capture a screenshot or video clip. If a video clip was necessary, then the clip needs to be edited and posted to a clip sharing site. Then that needs to be added to the report and I always check to ensure the clip can actually be played. I have to look up version numbers for a build (if I’ve neglected to memorize it) or the version number for an official aircraft or other official add-on.

I know that many other users are investing similar amounts of time/effort in reporting bugs, too.

All of that takes real time – time away from actually flying and enjoying the sim for pleasure.

In exchange for that personal investment, I would like to see the bug management on the forum be better.

I have definitely appreciated the forward movement of the management process since my initial days on this forum. The tags have helped. The evolution of the bug reporting form has helped.

However, it could be better.

I regularly trip over bugs in the sim that I know have been reported. These bugs are sometimes quite old and may pre-date the tag system or have even been reported after the tag system was implemented. However these bugs don’t have tags or any visual acknowledgment of their recognition as an Open bug.

Some of these have small numbers of votes. However, a vote count shouldn’t be a weight on a bug. A bug is a bug. In my experience we always assigned a bug priority. Crash bugs were the top of the pile, game breaking bugs would be next and so on.

Here’s an example of a bug that has 5 votes, is quite old (I saw this eons ago on Xbox) and has no tag. It’s what inspired me to write this post:

Here’s an example of a bug that just needs to be managed:

What is going on with that?

There’s a huge number of “bugs” like this throughout the Bug Reports section.

Here’s the thing. There are a number of threads here, on the forum, where people have listed off a number of open bugs/issues that are still in the sim that have been there a long time. I cannot imagine that MSFS 2024 is being built from the ground up, entirely from scratch, without using any of the existing MSFS code. If that is the case, many of these bugs are going to be greeting us at MSFS 2024’s launch.

At the very least, I would like to see the forum’s vast open bug list get a thorough going over. Some of the bugs might be mis-reports – user error, duplicates or have been fixed. Others might be poorly reported, with little detail and simply impossible to replicate. Others might be Open bugs, like the aforementioned, that have no acknowledgement.

I’ve looked over forums for other software titles where a CM or forum member has categorized open bugs and compiled a list of them. The weekly Development Update only lists 24 bugs by vote count. Well, what of the rest of them?

For the effort we’ve invested, I’d like to see a forum bug report synopsis where we can see the status of open bugs. Perhaps this sounds like a vanity project to placate Nixon Redgrave, but I assure you it isn’t. If these bugs aren’t categorized and prioritized how do we have any sense of their status on the internal bug fix stack?

I am not trying to be insulting in any way, but I have been surprised by a number of bugs that I have found that seem to have not been found by the internal QA team. I, clearly, have no idea how that team is managed and what or how they test, but if they aren’t able to catch the bugs I’m referring to, then the external volunteer testing team is extremely important to the process of catching and fixing of bugs in the sim. Given that is the case, there needs to be a more formalized process of managing those externally reported bugs.

I would happily take this project on. I know the forum is able to do some sort of reporting organization based on how it was used when we have done numbered release note testing in betas past. I would happily take on the job of going through all the open bug reports on the forum and flagging them for review as “not a bug” or “move to general discussion”, checking if they still exist, checking if they are intelligible, checking if they are duplicates, etc.

It’s a big job, but it needs to be done.

I know the CMs have a lot of other responsibilities as part of their job description. On the Friday Fly-Ins, I’ve heard them share what they are up to. I know they don’t have the time to be able to be here on this forum 40 hours a week. I’m certain there is a percentage of their overall time that they are to spend here, but that time cannot be completely devoted to the management of the bug list. Somehow, though, it needs to get done.

The more out-of-hand the entire Bug Reports section gets, the less effective it becomes as a tool.

I know there is an individual on the official MSFS team named Bugheadz, who very, very occasionally appears and posts. Is that individual doing everything I’ve outlined above? Do they need some help?

If all of this has already been done internally, then at the very least, all these untagged bugs need to be tagged, so we know our efforts weren’t wasted.

I know this simulator is a massive project with deeply complex branches of code. The management of the bugs that exist within that code is equally a massive project. As former QA, who managed QA teams and the bug database for an entire project, I am looking at this forum’s bugs and seeing a need to get them corralled.

I’d like to be a part of that process.

Thanks for reading this.

26 Likes
2

Hi, I can shed some light on some things you’ve mentioned

A lot of old posts are leftovers from when the forum’s bugs category also included help topics. Some topics like this one that are mentioning third-party bugs want moving out of Bug Reporting Hub and into a different category.

The Moderator Team has been actively keeping new bug reports tagged, tidy, and organised. An example of this is if someone creates a new bug report for a bug that has an old topic from '21 without tags, we will merge the new bug report into the old one and add tags, and ascalate the bug report for review.

Some Moderators (myself included) also actively trawl through old Bug Reporting Hub and #msfs:wishlist topics adding tags, merging, and moving topics to a different category where needed.

(You can see when we’ve done this in the edit history of the first post)

All forum members can help with this! If you see a Bug Reporting Hub or #msfs:wishlist topic that:

  • Is a duplicate of another topic
  • Shouldn’t be in Bug Reporting Hub/#msfs:wishlist , but should be moved to another category

Please flag these topics as ‘Something Else’ and write a short description of what you think the problem is. More detail is helpful but it could even be as short as “Duplicate [Link to other topic]” or “Third party bug” or “Not a bug report”.

As you said there are a huge number of bug reports to sort through, too much at this point for only one person or even a small team of people, but together the community can help to sort them out!

Cheers :sunglasses:

P.S. I moved the ‘Carenado Bouncing’ topic out of bug reports and will escalate the ‘propeller pitch’ bug

5 Likes
3

Does doing so count towards my limited number of posts that I am able to create per day?

I’ve already been very flummoxed by that limit whilst entering bugs I’ve found during the betas or when checking and doing numbered release note testing. Not to mention the frustrating “you’re replying too quickly” throttle when I’m clearly just trying to do a needed job.

If I have to flight with the forum to perform this needed service, I’m not going to want to do it.

3 Likes
4

I agree with your assessment that Asobo seems “list management” challenged. Not only for the bug list, but for the wish list. There are now actually two lists of asks from the user base: the official “Wish List”, and the “What We Want” list for 2024. These need to be combined into one list, organized by priority (impact on usage – not by vote), and made available to public. Almost all of these asks do not need to wait for the new “core internals” planned for 2024. The higher impact issues could be addressed before working full bore on 2024. “Nice to have” enhancements can wait until 2024 from my perspective.

I hope Asobo takes you up on your offer to organize and maintain the bug list, which will certainly be a consumption of your time! It would really be great if somebody would help Asobo in a similar way with the multitude of “asks” from MSFS users.

Since your post indicates you have been involved in the management of similar products (good credentials) I am curious about your take regarding what is considered a “bug”. Some of the items on the ask lists seem to be an overlap into the bug list. I guess out of frustration, users have put on the wish list things that Asobo developed and released, but don’t work well or at all.

In general terms, I would think a “bug” is something where code as been developed to provide XYZ, but when you try to do it, it doesn’t work. Then you get into well, is it just a bother, or some kind of user error (often the result of poor design/programming) or does it cause one of the four kinds of computer errors: wait state, loop, incorrect output, message. I would think another criteria for being identified as a bug is that it needs to be reproduceable by Asobo (otherwise it can’t be diagnosed). This issue of what is a bug versus what is an ask can be a grey area. If you can enlighten the subject I think it would help all of us appreciate the challenges Asobo faces even more (and give them some slack as they work on what is probably one of the most complicated pieces of software on the planet).

Here are some examples of that region of gray area between bug and ask…

  • Asobo provided a way to spawn additional windows that can be placed on your screen or on other monitors. However, from my perspective, they only went half way. The purpose of another window would seem to be to get another view (another camera). This would allow for a variety of uses, such as placing the instruments on a second monitor. But as it now stands you can only pan the camera of the main/core window. Is this a bug, or just a poor implementation?
  • Asobo provided a way to save and load flights, presumably so you could save where you are and come back to it at a later time. However this feature has never worked. It is so full of problems that you may as well just start another flight. Is this a bug or an ask?
  • Asobo put a lot of effort into creating a record/replay feature. Again, it is all but unusable. Bug or ask?

Thanks again for your generous offer to assist Asobo with bug management. I hope they realize how useful that would be.

4 Likes
5

This is a good question and good examples.

Since we are all “on the outside looking in”, we have no access to design documents nor did we sit in on meetings to decide how something was to be implemented nor can we walk over to the programmer who’s charged with implementing feature X and say, “Hey dude/dudette, this implementation is a bit kludgy, can we do X or Y or Z to improve it?” and then have a dialog about from there.

Quality Assurance is more than just identifying and reporting bugs, so we were tasked with writing up poorly implemented features and would suggest improvements to implementation and functionality.

How we do that here, on the forum, as users is tricky.

Example 1:
This seems very clearly an issue of what was developed needs to be improved, which is not a bug, but a feature improvement request.

Example 2 & Example 3:
These are a case, again, of poor implementation without any definition given to us, the end user, as to how it was intended to work. Many of the sim’s features have no documentation for us to refer to. For example, the Rolling Cache’s function and purpose has still not been clearly defined by an official source despite repeated requests from us.

I’d say the best that can be done is to write up the individual aspects of the Save/Load of flights and Record/Play feature that clearly break the ability to do so successfully as bugs. All we can hope for is a tag to indicate back to us if it is “feedback logged”, “bug logged” or “by design”. Then we can move the item to Wishlist status if it’s not a bug and hope for the best.

Clearly, we need some documentation explaining some of the core features of the sim, so we know what to expect of them.

2 Likes
6

I see what you did there :laughing:

I certainly can empathise about the ‘replying too quickly’, I’ve run into that many a time in the past (funnily enough while flagging duplicates!).

I don’t believe so, in the Discourse documentation limiting ‘Flags per day’ is a different setting to ‘Posts per day’ (however I sense you may also run into this at first :stuck_out_tongue_winking_eye:). Unfortunately I don’t know exactly what the afore-mentioned limits are (they are Admin settings, so CM etc.), I only know that they do in fact exist in some form or another

2 Likes
7

HA! I didn’t even notice that typo! Freudian slip??

1 Like
8

Maybe it would be helpful if Asobo and Microsoft would sometimes show a few screenshots of their REAL bugtracker? (Not the current voted list in the snapshots). Perhaps they have a different (better?) category system?

IF… if all of the effort that dedicated and capable testers expend is as valued as they say on Developer Livestreams, then maybe show us a bit more of the real tracking process? That would provide a bit of reassurance that user reports aren’t just collecting dust in a dead letter office.

I also have another idea. I think Asobo should INVITE Nixon for a visit - give him an audience with the developers - as he has experience with both PC and Xbox versions of MSFS, experience in flight sim development, is so good at articulating issues, and has been dedicated to helping improve MSFS (and MSFS support) from a customer perspective.

1 Like
9

I agree that problem management needs improvement. Several things that may help (or may not):

  1. Have both a bug/problem severity and priority categories clearly defined. For example a severity 1 priority 1 means the program does not work for most or all users. If MSFS fails for me (severity 1) but only me, then the priority would be lower. (Of course, I want it priority 1 fixed right away before anything else is looked at.) The “votes” sort of give an rough indication of how many users are impacted.
  2. Users should be responsible for testing and validating the fix for the problem they opened either in or out of the beta.
  3. Social media has significantly changed users methods for discussing and problem resolution. This is an issue for most games, programs, and applications, not just MSFS. This forum is a communications focal point for users and the MSFS team. But not everyone uses this forum regularly. I read posts and comments on other social media platforms describing issues that either have already been fixed or a workaround is available in this forum. Also, some posts or comments may be for an issue not covered at all here. There should be a way to communicate with users who don’t participate in this forum. (Can Xbox app be used?)
1 Like
10

10-4. Thanks!

11

Hi,

Excellent post and I’ll add my thoughts to what I believe will help with better bug tracking;

  • Beta Bug Reports should not get Archived if the bug is not fixed; Something I’ve noticed during the Beta Periods is that when bugs get reported the whole section gets Archived at the end and people are unable to vote and continue reporting, the problem then arises is that many of these are never “Bug Logged” so you see a repeat post on the general bug forum post update release and therefore the bug gets “delayed” in being properly reported. This is not the correct way of handling bug reports during beta periods, in fact IMO the whole beta period needs a complete overhaul as it’s clearly not working correctly at all (from my experience in software and also QA) but that’s another topic entirely.

  • I totally agree the “Votes” system for Bugs is flawed, for wish list items yes it’s great but as you say a Bug is a Bug and there are some very old and very immersion breaking bugs that are yet to be fixed and the question has to be Why? It’s it because they don’t have enough votes? I would sincerely hope not but I still have to question why.

  • Direct Communication; It needs a developer on the team to actually respond (especially during beta periods) to Bug Reports and issues, it’s very clearly not working when Moderators are the “go between” and forwarding on issues, just look at the bug lists and again how many of them are so old, are they even getting through to the dev team? Or are they only doing so when they hit a certain number of votes? No one knows because there is zero communication (again this point goes towards the above Beta Period Overhaul). It simply must be improved, those of us helping out during beta periods, collecting the reports and so on would feel like their contributing much more if the actual developers were on hand to spend - just a few minutes - responding to the more serious bugs at least. I’m not asking for them to spend hours on the forum, responding to every post as that would be just silly. But if they’re going to post anywhere it must be on the beta forum, again being in this industry I have been on and used my team’s effectively engaging with the public during our own beta periods and it helped us immensely as well as the users who felt part of the team when helping out. This way those bugs get through directly to the team in a better format than relying on other people to forward them on, looking across this forum since it’s Inception it’s clear to me this is a major failing.

Thanks

4 Likes
12

Good post @NixonRedgrave

Agree with all written here. Mentioned myself similar things (in less words :stuck_out_tongue: ) a few times lol

I think the key thing would be just to have a more direct link to the studio level to respond to some of the points. The more serious core ones. Not every little thing of course as that’s impossible.

I know they give some good thorough answers in the developer forums. Eventually. And it’s done with detail and intent when they finally look deeper into an issue (example: the problem Burning Blue Designs have with ground textures from Bing taking priority and showing through if user has more than around 8 custom airports in a region… of an undetermined size. Problem replicated internally now but they say it’s so deep into the system that a fix is unlikely in the short term… doh! But at least we KNOW. Not just in limbo land).

But without that contact back to the reporters, we are in a black hole void of uncertainty.

It feels, to me, without that contact and response back, that the whole thing is in chaos and the ship is sinking. That’s not to say MSFS is failing. Of course not. We all love it and that’s why we put up with the issues and spend so much time trying to get things resolved instead of flying for fun (half my flight time is testing and fixing I guess).

Maybe a better analogy is that the people in the engine room are drowning… a bit? :ring_buoy: :ocean: :hook: :angel:

Perhaps there should be an effort or push to allocate in hard terms a set number of hours per week - maybe like half a day one day per week, for someone that has contact with all relevant programming departments to compile a list of key issues and get the answers, then respond in an official manner here publicly. Tagged.

Not just about bugs but as written above, answers to HOW things are MEANT to work like the Rolling Cache. I’ve also specifically asked about this with no response. Into the black hole it goes!

I don’t mean to be negative here. I hope it doesn’t come across that way. Just see there is room for improvement and inclusion that would go a long way to appeasing the situation.

Especially with the upcoming future of the sim in 2024, it would be good to get the house of cards in order with a solid ground to work from and keep everything / everyone on board. That’s enough of the ship analogies from me :wink:

1 Like
13

There’s lots of good feedback and ideas on this thread.

This one in particular is really bad. If a bug is raised during beta, and not fixed during beta, then it should be promoted to the main forum and not archived. With very limited exceptions bugs don’t vanish at the end of beta. If it didn’t get fixed then instead it’s now shared with everyone.

2 Likes
14

This is a great thread. I’m learning a lot.

I’m wondering how effective the “vote” method is for prioritizing developers time (which is finite). Because MSFS is used in so many different ways by different users, what’s important to some may be considered a “nit” by others. Also, how many MSFS users** are on this forum and take the time to vote on bugs or asks. I know I don’t vote on bugs (but probably should). Is vote count an accurate indication of a particular issue’s impact on the usability of the product? To me what would be a “show stopper” issue, and which I’ve posted in the “wish/bug list” may only get 5 votes because most others don’t use/need that. I mainly fly pre-1950 SEL planes VFR for the purpose of sight seeing, aerobatics, and video creation. This is pretty limited usage compared to others. I am frustrated because I’ve had to resort to 3rd party solutions (which I try to avoid) when the capability has actually been implemented in MSFS, but is so poorly done that it’s unusable (resulting in thoughts of “what were they thinking?”). So I put suggestions for improvement in the wish list only to never see them appear on any Asobo list (because of lack of votes?). But to get back to my point, is there a better way for Asobo to recognize and prioritize the “perceived needs” of the majority of users? Hopefully not a committee – we all know about committees! :o)

** How many “users” are there? Web searching has not turned up a clear answer.

4 Likes
15

I think votes for features or wishes is fine. It gives at least some sort of pointer to what people want. Even if the sample size is low here, you could argue that relatively speaking the ratio could be scaled up to the entire player-base fairly accurately (…and no, I also have no clue what the active player-base really is now! Or where to find out).

But as you and others have said here, votes for bugs really should not be the determining factor as to if it gets taken seriously.

I do believe that some of the issues are just so ingrained in the layers of code from the beginning that it’s next to impossible to iron out now. Hence them having to rewrite a lot for 2024. I have faith that this will indeed solve some of the long standing issues. But on the other hand I’m not holding my breath.

Even with the “black hole of conversation” I’m pretty sure they are well aware of most of this stuff and it’s getting through to them. Just the gravity in that black hole is so strong with all the bug-matter that’s been thrown in that it won’t let any communication (or light, or electronic bits of data) escape back to the forum!

3 Likes
16

Excellent way to put it!
:+1:

1 Like
17

It’s been concerning, when on a Q&A session, an issue is brought forward that has been very visible on the forum and the panel seem to be caught unawares.

It’s these issues that seem to slip under the radar that concern me.

A “bug-logged” might be a placebo, but I’d rather that than see absolutely no tag at all.

5 Likes
18

I hesitate to add anything more to this great conversation, except maybe this:

I think because MSFS is an experiment in progress, with a massive task, massive audience, and many teams working on separate aspects (siloed) - sometimes what the developers need is how it all looks from customer perspective. There are some aspects that are very detailed and well thought out, and others that are just legacy, “procedural”, or neglected and not prioritized.

Regardless of the “Level” of the sim as a whole - I’d like to see QA ensure more things be brought to the SAME level, for consistency. I would call all of the issues / bugs (with the sim, and with support) a “Problem of Success” - when you create realism and wonder so well, then anything that isn’t at the same level disrupts the experience and stands out.

If it is just left to individual users reporting and vote counts, then that type of QA doesn’t happen. The resevoir is a gold mine for QA - they just need to capitalize on it better. Maybe they are…

1 Like
19

Exactly. Resulting in a lack of confidence in leadership awareness.

4 Likes
20

Hi @NixonRedgrave,

The bug-logged tag is not a placebo, we have a process that is conducted to ensure the correct bugs are assigned to the correct team internally. When these bugs are logged in the internal system we also make sure to update the forum thread to reflect this stage.

I’d disagree that issues brought forward on the forum slip under the radar. Your perception that members of the panel are caught unaware may potentially be down to the wording of the question if written by a viewer in the Twitch chat, or whether that member of the panel is actively working on the topic at hand. There are many people working behind the scenes on a variety of different areas.

As I’ve mentioned to you personally a number of times now, we take an enormous amount of time looking at the forums alongside a range of other platforms in order to gauge feedback from the community. There are different ways we engage with these topics, be it a reply in a thread, asking about it in a Dev QnA livestream, logging bugs etc. As also mentioned, we aren’t able to reply to every single thread that appears…but we do our absolute best to at least read every single one! :slight_smile:

We continue to strive for improvement in all areas, so constructive feedback is always welcome.

Thanks,
The MSFS Team

1 Like