Is all development this flawed?

Since I don’t have any experience with games or software development, I am very curious as to how things work with updates and development.

In my experience with MSFS thus far, every patch release has fixed a few issues, but also created new ones. The latest patch seems to have broken the way liveries are being read and although Asobo can’t seem to recreate this issue my sim looks way worse than it did at launch with the exact same settings. Up until a couple days ago I had zero stutters but now every time I fly I get multiple lags which can last a few seconds, this seems to be in conjunction with the server side fix with weather they mentioned in the QnA.

I don’t game or develop software, all my experience is with Flight Sims and music software such as VSTs and DAWs. Since I have no other point of reference I want to ask others if this type of issues are normal or is Asobo doing something wrong? Have they taken on more than they can handle?

Keep it a healthy meaningful discussion please.

3 Likes

It is very complicated to answer this question because the game itself is very complicated. The game is an order of magnitude more complex than your average game, for many reasons:

  • the visual algorithms for natural elements are state of the art
  • it does portance dynamics and perhaps fluid dynamics in real time, this is very CPU demanding
  • the game simulates the complex electric and electronic systems of aircrafts, which are all different
  • the game depends on the Microsoft Azure infrastructure
  • it depends on data coming from many different sources (geographical data, meteo data, traffic data)
  • it’s massively multiplayer

And ALL OF THESE elements are combined altogether, and all of them interact with each other in complex ways, and this must run at 30 or 40fps. This is why simulators are much more complex than most games. In most simulators, just simulating exactly the behavior of one single plane is a full year or a 2 year project, that’s why they are often sold more than $50 for a single plane!
So these are some of the many reasons why the game is hard to test and somewhat unstable. Plus there are lots of 3rd party addons that can crash, because the game is still in beta and the addons themselves are constantly in beta phase.

Was this project too big for Asobo ? I can’t say, they pulled it off and I don’t think many companies could have. But otoh they seemed to consider testing their product was optional, which led to 2 very bad patches. I think with the last 2 patches, they seem to have learned their lesson and things are going in the right direction.

12 Likes

This is a common thing in (game) programming.

IMHO Asobos handling seems a little rusty, but ok they are dancing with microsoft. All in all they are doing good with the problem (program) at hand.

2 Likes

Sometimes the kind of state that the code apparently is in right now is caused by higher-ups telling the low-level coders “Good enough!”, when the coders really would like some more time to make sure everything is working properly. A long enough period with “good enough!” quality will eventually lead to huge interconnecting issues that are difficult to fix.

And then there is the pressure and expectations from the publisher to deliver, because marketing has already portrayed the product as something that in reality is not there.

4 Likes

MSFS is no different from other games. The bigger the game the more bugs you have. The problem is that asobo has a schedule and they were pushed to release the game a bit early. But overall the experience is great so far. didn’t touch xplane 11 since MSFS came out. It’s just so much better in every way.
Patience, in 6-12 months most things should be fixed. And if you expected a live-service open world game to be bug free on day one you either never played a game before and it’s your first or you knew it won’t be perfect and you’ll need to wait for updates but still bought it.

I’ve been mainly lurking here for the past month and am glad to see that someone has asked this question. I am an engineer who designs systems for business jets. I have also done firmware development and software development. I have never worked for any truly big companies or on any truly massive projects, but I can tell you a couple things for certain and also a relatively interesting story that might shed some light on the situation.

While you’ve almost certainly used a self-checkout before, you probably have not considered how much they themselves cost. A large self-checkout used at a major grocery store can cost, in low volumes, something around $30,000 each. Obviously there’s a lot of hardware there but there’s also a fair amount of software. I worked for a company that was developing one to be much smaller and much cheaper, perhaps $3,000-$5,000. When I started there were two developers, both of whom had worked there for something like 5-7 years. They had been working on this self-checkout for about 3 years at that point and had not managed to get the core software into what I would consider a stable alpha. It wouldn’t scan but then it might scan ~100 times in 5 seconds. Transactions wouldn’t get sent into the cloud. Transactions wouldn’t get authorized correctly and would either not charge the customer, or, if the scan bug was “triggered,” it might charge them all 100 times. The payment processing back-end would go down so often that we would joke that it was running on a laptop that someone left out in the break room.

This was after about three years with two “mid-career” developers working full-time on it. Did I think they were horrible? Yes, absolutely. Did I think I could grab a reasonable motivated intern from a local university and get a minimum viable product done in a week or two? Yes, and I suggested it. But were they the only ones in the equation that were even close to being this bad? No, absolutely not.

That self-checkout was demonstrably bad. Little to no architecture, implementation that would make any .net developer cringe and most people in the open source community have an aneurism, and a fundamental lack of knowledge around… well, software, I guess. It seemed like it was maybe on the order of a weekend woodworking project from someone who saw an episode of This Old House and then decided they were going to build wall-to-wall bookcases with what they had in their garage. Except they were on payroll getting paid significant salaries with what appeared to be almost no practical, let alone theoretical, knowledge of what they were doing.

Wikipedia is telling me Asobo Studio had 140+ employees in 2018. Wikipedia says that MSFS 2020 was announced at E3 on 19 June 2019. The game was released on 18 August 2020.

The complete software package at the time I left the company was about 100 MB and was maybe some 10,000 lines of code with the vast majority of the installer containing the “framework” required for Windows to execute the actual thing. MSFS is around 1200 times larger and, if I had to guess, is comprised of tens of millions of lines of code. I would guess it’s almost exclusively written in C++ for speed which is not often considered easily approachable. The stuff that makes the displays and MFDs work is not just its own code but an entire engine they built and ported in so that mod developers could use the type of tech used for the web to make MFDs function instead, presumably to make it more accessible for the much, much larger number of people who have done web development and not relatively low-level C++.

An artificial intelligence took geospatial information and satellite imagery and, from what I understand, with minimal human interaction transformed a 3D ground layer and a 2D image and mapped that into a game with the “level map” being 196.9 million square miles (510.1 million km^2) in size. Skyrim is 14.3 sq mi. Someone on the internet is saying World of Warcraft is maybe 200 sq mi. Another person is saying 61 sq mi for WoW based on a different measurement method.

That is ■■■■■■■ wild.

But then, as if making an entire planet into a playable game wasn’t enough, they also needed a physics engine to actually make it a simulator. I’d be shocked if they didn’t get in touch with computational physicists and particularly people specializing in fluid dynamics. Accurately modelling the real world is, as you might expect, kind of complicated even if you hand-wave away a lot of stuff. I’ve never done anything like it personally but it seems like a field where first-approximation results are relatively easy to hit but very quickly becomes enormously more complex (i.e. modelling a smooth sphere in an atmosphere isn’t particularly hard. Modelling a golf ball is a great deal harder. Modelling an airplane is probably orders of magnitude more complex and hints of it are shown in the flight_model.cfg). Even the statement where “well it works in FSX/P3D/X11, why doesn’t it work here” falls apart quickly in my mind if I consider that I don’t have a clue what they accounted for in the flight models or simulation code of any of those or what they’re trying to accomplish here in MSFS 2020. I would expect this to be quite a bit more ambitious as the specs of FSX are maybe less than what a netbook has today (1.0 GHz processor, 256 MB ram, 14 GB HDD, 32 MB DX9 graphics card).

All of this coming from maybe 140 - 200 people in France, at least months of which was during a worldwide pandemic and recession, each with their own lives, families, problems, etc.

If this development timeline is even close to what is implied by the announcement and release dates, I would not have bet literally any money whatsoever that even an experienced dev team could release a game of this scope in a year. Even three years I would be deeply, deeply skeptical about.

So where does that leave us? For me personally, I picked it up on game pass for $1 first about a month ago. It looked like, and still largely looks like, an MVP: minimum viable product. They’re operating on what appears to be 2 week sprints which I would expect is lagging a week as one week is probably for development and another for testing before release. I’m certain they have dedicated QA staff and playtesters but, as we’ve seen here and everywhere else in the software development world, it’s possible for someone to test the ■■■■ out of something and pass everything on a dev box but then as soon as it’s in production bugs show back up or were never removed or now appear differently or whatever.

I’ve already written a whole novel at this point but if you really want the tl;dr version I would say this: software engineers are not exceptional. There are just as many, if not more, bad software engineers than there are, say, bad carpenters or bad doctors or bad babysitters or bad teachers or bad cashiers. If anything, the complexity and immaterial nature of the work can obfuscate and conceal flaws in people’s reasoning or understanding and it is an exceptionally difficult field to hire in. The fact that they delivered a game literally at all even remotely close to the stated goal is, in my opinion, a testament to that team’s abilities. I would not think twice about asserting that these are some of the top people in their field.

I gave them my $100 after I saw the bugs and what they had accomplished, for whatever that’s worth.

35 Likes

Agree… :+1:

5 Likes

Couldn’t have said it better. The magnitude of this undertaking is nothing less than genius. To have the entire globe to fly around in is a miracle in itself. And OSOBO is committed to making this right over time. I got a little bored with the genre for a while. But, now you add exploration in a real world… I think this is fabulous and deserves our patience. THANKS OSOBO… we aren’t all out here wining.

10 Likes

Might be good to acknowledge them by name too…

2 Likes

I really agree, in open source they work really more efficient and…may be the most important part:
they listen to the users and, by teh way, motivate them to become part of the project

well you signed it: this is more a moloch than a geme. Sure there her are tons of beatiful pictures, but they are not even able to assign a geographic level correct…I reported that: in Landing challenges Kemara is assigned as 6451 ft Rnw 19 (like Jackson) and it is in Reality: 160 ft Rnw 20…
I think it´s just poor and just a thing besides, but…

they fill us up with bytes, just to show us: the more the better…the usual way business works…together with these hardwre people

I really appriciate your statement about human power, the facts (not fakes) about the differece between the dealer (Microsoft) and Asobo. POINT

after using Microsoft (more sometimes) for more than 30 years I would say:

I gave them my $100 and didn´t expect anything else as what happens here right now…they will not care…

it was pretty smart of them, not to do it themselve…

Flightgear does everything of your points…exept the point about <microsoft

so what

Asobo is WAY too small of a company to pull something this big off. Why? 1) Look at their previous titles. 2) still aren’t fully staffed up looking at their available job postings, 3) Look how the initial release went… Why Microsoft picked them is surprising. I hope they grow and continue to support the title but I have my fears now.

2 Likes

Having seen lots of software (houses) from the inside over the years … I have to say:

I am actually deeply impressed with FS2020.

So I feel like I totally agree with @Mienyn here.
Given the mind blowing complexity of this entire game + backend … and the incredible scale (users, data volume, etc.) … I would really be interested in finding out, what their “magic sauce” was, to pull this off.

I am most fascinated by the fact that besides that one “Xbox controller crashing the game on device sleep and wake” issue (which they fixed in one of the first patchs) I have not experienced real (hard) crashes (OK … I did crash lots of aircraft … but that was me … not some use-after-free or zero-pointer C bug).

The lack of crashes makes me think, that they must be using some clever system design (e.g. lots of WASM sandboxes?) to isolate different parts from each other. Especially where third party code (e.g. from aircraft and airports) gets included.

Sure, I do run into lots of bugs, and as anybody else (and I assume this does include the people at ASOBO etc.) I would love things to be “better”.
But I am here for the “long ride” … and I understand that the first year(s) will be rough.

5 Likes

Yes, you better believe it!

I purchased the program on the first day it was released. I was quite impressed. Now after the last updates the screen colour is washed out, on take off the screen freezes roughly three times, the A320 autopilot has faults, the B747 has a mind of its own on approach and the weather is not live. What is going on? It’s now far worse that the original.

2 Likes

The screen washed out sounds like you have DLAA enabled, be sure to turn if off if so. I don’t even know what that feature is for, it looks awful.

Yeah like I’ve had some buck wild issues in the past 40 or whatever hours:

  • logbook thinks every flight is a past flight (theory: something was not being initialized properly and/or the record got stuck in some sort of lock file or active session file and would just get read every time instead of being written to)
  • logbook doesn’t record flights (again, seems like a record isn’t getting written but I have no idea why)

I got both of those to stop happening after this last update by signing out within the app and disabling data, closing, and reopening. That worked for a time but it then failed again and I haven’t been able to recover it.

  • Graphics performance is… interesting. As in my other post, it looked like it took a weird step backwards in 1.9.5 for me as others seemed to be saying it got better for them. Framerates dropped, CoherentGT went through the roof, no idea why. Current version brought framerates back to ~50 FPS at 1440 on ultra but the stippling is very pronounced. Others are reporting their graphics are worse again.

  • I’ve uninstalled and reinstalled 4 times so far and have been somewhat baffled by them leaving in the overhead of transmitting individual files not to mention no way of downloading the installer. Having gigabit fiber, though, is probably the only reason it didn’t drive me crazy.

I haven’t had any CTDs even though I was on Windows 10 2004 and now 20H2. I had several installation and login issues with 1.9.3 or whatever was in prod a month ago.

I could go on but I think it’s fair to say that everyone in game, and especially the people who give a ■■■■ enough to go on manufacturer-hosted forums, have had varying degrees of frustration. I think that some of the frustration is not despite but because of the potential here. I guess the only other perspective I can offer is that it looks like MSFS has released more patches in the last month than I believe Nier Automata had in its entire lifetime despite fans practically begging Square Enix for any patches at all. NA wasn’t exactly a cheap title to produce and both Square and Platinum have plenty of experience in the field. There was no reason to abandon development but they did it anyway for reasons that elude even the game’s director.

While I’m sure Asobo and Blackshark have plenty of talented people, they surely have key players who have been absolutely integral to the development of the project. My only “fear” (if that even makes sense in this context) is that key players who developed and/or are expanding the technologies get poached or otherwise pulled off the project. Most likely, though, I would expect that they all signed contracts, NDAs, and/or noncompetes plus they’re working in the EU where I’m unfamiliar with how that would work.

1 Like

Gonna add my $0.02, when you see other games out there that use brand new state of the art engines, there’s always a load of bugs that make it unusable in some instances, and others that cause the games to CTD. It isn’t until after a few high profile games make use of this engine that the engine stabilizes.

And MSFS has multiple state-of-the-art engines running.

My thoughts: compared to the rest of the gaming industry, MSFS is on par (arguably slightly better) with other groundbreaking engine games. The only reason we’re seeing so many bugs is because we’re seeing 3 engines, that would be developed over the course of 3 years, in 3 AAA titles, all being developed in the same game within 3 months, so issues are far more glaring than they would be to the average gaming user.

3 Likes