Why is there no Backup plan for server failures?

Hi,

Since I bought this simulation earlier this year I have noticed it is plagued with server related outages, whether this comes from Microsoft or external partners who knows:

  • Live Weather; down most often and clearly a serious ongoing issue that needs immediate attention.
  • Azure Text to Speech; Had this happen a handful of times this year which the on-screen dialog.
  • Connection messages in game; even though my connection is perfectly fine I’ve had these a few times as well.
  • Incredibly slow download speeds when updating; every single update practically, a severe server issue during peak load.
  • Bing Maps; first time today that I’ve witnessed but another example of a server outage.
  • Tile Loading issues during flight; a big issue noted well over a year ago which could well be a part of a server related issue.

What troubles me is there is seemingly no backup plan for these outages, for sure not every server related issue can have a backup plan but most of those can. Take Live Weather for example, the recent issues were down to lack of METAR transference, so why can’t the simulation use Meteoblue as a back up to use in these times? The Sim used to rely on Meteoblue 100% before SU7, so the knowledge and know how is there to be used.

How will the next sim manage to deal with extra server load with it being an even more predominantly server based streaming platform? What protocols, procedures, backup plans are in place for when that arrives?

These server related issues appear to be getting worse before they are getting better, and that is worrying as if these happened during a holiday weekend they might not get fixed for days. A few hours here and there may be ok for some, but they’re adding up to show a serious problem in the pipeline overall.

Thanks

5 Likes

The Sim used to rely on Meteoblue 100% before SU7 ?

Can you explain that statement, and any official source ?

things break. That’s the nature of technology. These things you listed have broken at one point or another, some multiple times, but they get fixed within a reasonably short period.

There is an entire industry of IT dedicated to disaster recovery. Personally, i would not share my DR plans as a matter of security but it would be stupid for an organization as large as MS and Asobo to not have an IT team who sits around waiting for something to break so they can investigate and resolve the issue. In my experience, 90% of fixing an issue is trying to identify the issue itself, more than just “yup…it’s broken and doesn’t work” if that makes sense.

That is, you can safely assume there is a plan in place by the MS and Asobo IT guys when things go down, probably based on some industry standard for disaster recovery tailored to their needs. If it were as easy as “switching over to the backup servers” then this thread would exist…

Hi,

I was referring to the time before the METAR integration of SU7.

Thanks

Hi,

Precisely my point, things are “breaking” far too often and what if those short breaking periods become much longer? There is no backup plan if that happens, planning ahead for the worst case scenarios is software coding 101 when dealing with outside influences like here.

Thanks

1 Like

Hi,

There very clearly isn’t, hence this thread and again in my short time of being a user I have experienced a bucket load of server related issues, not to mention the ones I haven’t but other people have. That is the concern here I’m showing here and it’s validated is you look back and read all the times there’s been server issues in some way or another, more so than any other game/simulation I can think of in the past 20 years since streaming became more widely used in the industry.

Thanks

This was one thing that stood out to me with the FS2024 announcement, and I share your concern on this item specifically.

I don’ t have a good answer to the overall problem. There are so many unique points of failure in the whole chain from a storage devices in some server rack in a building somewhere in the world to the network port on your PC, I fear there’s really no great solution other than “improving it all”

4 Likes

why are you creating multiple replies for something in the same post?

At any rate, it’s naive to believe there’s no plan in place for outages for an organization as large as Asobo/MS and an application as complex. For all we know, it’s entirely external factors beyond the control of asobo and MS.

Just because the developers haven’t publicly disclosed disaster recovery procedures doesn’t mean they don’t have one. I personally wouldn’t due to security concerns and the flight simming community is a bunch of…welll i’m not gonna go further there but we can say you just can’t please anyone these days…

Where I’m at there’s over 70 data centers with at least 10 going up now, all on a nuclear power grid. Maybe they should rent some space.

Before the sim was released, I thought the sim would have an online mode with streaming scenery plus an offline mode that would be the FSX style of scenery. Airports, Airplanes, and sceneries would all work in both modes and I was sure that Orbx would continue to sell their updated textures like they always did. I figured most people would choose the new streaming online scenery world because of its realism.
The offline mode that one can choose to download on XBox is about 60Gigs iirc which is much larger than FSX was and so I’m not entirely sure why they did what they did unless its just impossible to use the FSX style texture system as a fallback to the streaming texture system.
Maybe they wanted a complete break from FSX but ,if so, why have legacy flight model?
I’d agree they need better backup plans.

Hi,

We are all entitled to share our thoughts and concerns with the title as customers, if we didn’t there would be little progress. Therefore please respect other people’s opinions even if you don’t agree.

Thanks

METAR integration began in SU2, way back in late 2020. That was when they began using METAR for airport surface wind, temperature and pressure. SU7 brought the addition of METAR clouds and visibility.

The 2-3 hour METAR outage last week was very unusual and outside the control of MS. It was an outage of the WMO worldwide data feed that supplies METAR data to all r/w consumers of weather data. When it happened, even official government weather sources such as the FAA’s https://aviationweather.gov had no METAR or TAF data for any airport.

I assume that MSFS has an internal list of all worldwide airports that report METAR data, and Live Weather is structured to use that data for sim users that have spawned at an airport that reports METAR.

I assume that the only time MSFS wouid use the MeteoBlue model for airport surface weather is when the airport does not have METAR observations, is not on the “list”, and there are no METAR-reporting airports within several hundred miles whose METAR could be “imported” This is the case for quite a few small airports in remote areas of Asia, Africa and South America.

Asobo has done a lot of work over the past two years at blending METAR data and the MeteoBlue model. It works much better now than it did when SU7 first came out, though there can still be issues. It is probably not easy to have an automatic fallback in place to ignore all the blending algorithms and revert to MeteoBlue surface data everywhere in case of a worldwide outage of METAR data from official sources, because such outages are extremely rare.

It is probably easiest and safest for the sim to revert to ISA standard weather of clear skies, 15C temp, 29.92 pressure and zero wind at an airport that should be reporting METAR when that data is missing.

2 Likes

Since you seem to know about this, can you clarify if the MSFS user is getting weather data direct from the MetroBlue server systems, or from MS servers, that are themselves getting it form MetroBlue ?

Bluntly, whose servers are going down (and need to be fixed), when we get lack of weather , (Or old weather) , MetroBlue or MS ?

Going to be interesting with 24 being more server based

I’m 90% certain that the MeteoBlue model data does not come directly from MeteoBlue. They are not in the business or capable streaming to thousands of connected sim users. They create the model, send it to Azure (where the data may have to undergo further processing for use in the sim), and Azure streams it to MSFS clients.

METAR data might come from MeteoBlue - but I suspect it probably comes from NOAA.

2 Likes

There is a backup plan use a backup simulator. :laughing:

But you have the FSX style textures as backup. The game runs (and looks bad) when offline.
The bulk of those 60-200GB are the textures for all the 3D objects like airports, airplanes, POI’s etc. Those are completely offline (to be changed in FS2024).
Currently the online streaming part, as far as I understand, is only for the ground textures, 3D mesh, object placement and photogrammetric objects (cities). So the online part will tell the game to put house #97 at coordinates X,Y,Z but the model and textures of house #97 are already downloaded in the base game or as part of the world updates. I believe this is mainly what they want to change in FS2024 where also the textures and models will be downloaded. This is mostly aimed at the XBOX crowd with it’ small hard drives + it makes deployment of world updates easier, because you don’t have to distrubute the whole package to everyone. Just stream what is needed at the moment.
Off course this will in effect make the game even more dependandt on the performance and availability of its servers.

I wasn’t aware FS20 used the landclass lookup system. I wonder why Orbx hasn’t released any of their global texture upgrades? That would certainly make offline a bit more appealing. I thought it was just low resolution satellite imagery or imagery faked by Blackshark AI.

How many people would buy it?
Arguably the biggest reason for MSFS’ popularity is the real world ground imagery instead of the landclass textures we had for over 20 years.

Not enough people to justify Orbx making a new compatible landclass add-on for MSFS, IMO.

1 Like

Either it uses the FSX style landclass or it doesn’t. If it does (as I was told above) then the textures already exist and they would just have to be ported over as some have complained about some 3rd party addon airport developers doing.
As for the market, I think it’d be great to have a fallback (for when servers are down) that we could choose to dress up a bit. I thought the Orbx Global landclass was quite a nice product for its day and would rather have that than what I’ve seen the Blackshark fake imagery look like. Maybe its just me.

1 Like