SimConnect API Protocol

In another post, I postulated we could bypass the SDK completely and connect directly to the FS API over the network, if only we had some documentation about the protocol (comms format) the API uses.

So, I did some more sniffing about this, in another forum (oh, the shame of it) and it looks like the author of jSimConnect has done exactly that.

Unfortunately, his original post is no longer on the forum, however, it does still exist in the web.archive site.

Armed with this initial information, I plan to develop a proxy to capture comms to/from the API, possibly whilst capturing the actions being requested via the SDK.

For those who aren’t sure about the difference between the SImConnect SDK and the SimConnect API, let me explain.

The SDK is the DLL provided to developers to embed in their own applications, to communicate with MSFS2020. It is a proxy for transforming your requests into a binary format (protocol) suitable, for MSFS2020 to receive, and to transform the responses back into data the client application can use.

The API is the network endpoint in MSFS2020, that the SDK talks to, to transmit requests to and send responses back. It is an interface into MSFS2020 itself.

I plan to create a project in C# (which I will put on GitHub for others to access/review) using the SDK toolkit I created, so I can force the SDK to connect to my proxy, submit requests and capture the traffic between the SDK and the API, this will allow reviewing the traffic.

Using the initial documentation linked above, we should be able to enhance the documentation with a view to develop our own SDK’s, in any language, opening the door to so many projects, in almost any language, on any platform.

8 Likes

Python - SimConnect: GitHub - odwdinc/Python-SimConnect: Python interface for MSFS2020 SimConnect.dll
And a neat usage of that: GitHub - hankhank10/MSFS2020-cockpit-companion: Web wrapper for reading and writing data from Microsoft Flight Simulator 2020

Some docs on Events and Sim Variables: Event IDs | Microsoft Docs and Simulation Variables | Microsoft Docs but obiously for ESP, so a lot out of date

Thanks @DNBOF , I’m not too familiar with python but after a quick look at the repo, it looks like that project is still using the SimConnect SDK (DLL), my intention is to bypass that completely by accessing the API directly.

Also, please don’t forget the “original” SimConnect SDK provided library (and the one the documentation is based on) is a static library, not a DLL. The DLL is a later addition to help people wanting to use “Managed Code”, aka languages running on the CLR such as Managed C++, C#, and F#.

It’s been a while since I started this thread, but I’ve actually been doing quite a bit in the background.

I’ve created a small Proxy application that can be used to intercept communication between the SimConnect SDK and the SimConnect API (MSFS itself).

To use it, you need to reconfigure your SimConnect SDK to direct traffic to a different Port number (I used port 499 instead of the default port 500).

This is fairly easy to do, just find the SimConnect.cfg file on your computer. Usually, this is in the same folder as the SimConnect.dll (SDK).

It will likely contain several connection options, similar to below, just edit the local connection and change the values, as shown below

Protocol=IPv4
Address=127.0.0.1
Port=500
MaxReceiveSize=41088
DisableNagle=1

Launch the Proxy application and click the button to start the proxy listening on Port 499.

Run your application that uses the SimConnect SDK, it should connect to the Proxy application, which will then immediately attempt to connect to MSFS on the normal port, 500.

Thereafter, all communication between the 2 will update the Sent and Received boxes, in a hexadecimal layout, for you to examine.

You can stop the Proxy and all Sent/Received info will remain. However, if you start it again, both boxes will be cleared.

The Messages box does not clear when restarting the Proxy.

All boxes will retain 1,000 lines of text, to prevent you from filling up memory with old/stale data.

I may add a feature in a future release to save the Sent/Received data to a local file, for offline review, we’ll see.

3 Likes

I’ve added a feature that will also save all sent and received data to a local CSV file, so it can be easily imported into a spreadsheet for review

And so it begins…

For anyone who wanted to communicate with MSFS directly, without relying on the SimConnect SDK/DLL.

I have started creating a C# Open Source project that should work on all platforms (Windows, Linux, Mac, Android, iPhone, etc) that will allow greater flexibility to communicate directly with the MSFS API (that’s the network connection provided by MSFS, which the SimConnect SDK actually connects to).

I’ve already managed to break down the first request, sent to MSFS when SimConnect first connects.

Also written the initial “connector” application that sends/receives data to the API.

Early days but we’re on our way to demystifying the API Protocol and allowing us to bypass the SimConnect SDK completely.

That’s not intended to be offensive or derogatory towards the SImConnect SDK or its developers in any way.

The SimConnect SDK will obviously still exist and be used by many developers to come, this project is to provide an alternative for those situations where the SimConnect SDK can’t be used, as an Open Source solution for others to port into their own development language, or for when the bugs and limitations of the SimConnect SDK can’t be overcome.

2 Likes

As a shameless shout-out to any other developers who want to get involved, please message me.

The more of us that work on this, the quicker it will be usable by everyone else and the faster we can fix any bugs etc.

Lets make this a truly “Community Driven” solution for all of us to benefit from!

3 Likes

That sounds interesting, especially the “platform independent” part: currently people are writing their own “network protocol wrappers” around the SimConnect API, in order to access the API from e.g. Linux. E.g. Little Navmap does that, to my understanding (I did not study that network code in detail though): you start a small “Little Navmap server” on the Windows / FS2020 instance and can then communicate via network from e.g. a Linux instance (where Little Navmap runs).

With your API that “custom network protocol / wrapping” would not be needed anymore.

However I do see a great “danger” here: while the SimConnect API is “stable” (“standardised”), the actual internal “network protocol” is not: so whenever Asobo / MS feels like they could change the message format at will. Which would of course require a new SimConnect.dll and each and every client app would need to (at least) re-link with this new SimConnnect.dll - but in the best case that would be a simple “recompile” (relinking) of the app, because the actual SimConnect API wouldn’t change.

And of course you probably need to write individual code for each simulation out there: e.g. Prepar3D might use a different network protocol than FS2020 (and if not today, then in the future possibly).

Sure, those risks might all be just “academic” (given the “speed” with which SimConnect evolved over the last decades - but that could change with the popularity of FS2020 quite quickly again).

Besides that I am not quite sure why one wanted to absolutely avoid the SimConnect API itself: sure, it feels dated. It is not object-oriented. But it is written in plain C (with glorious structs and #DEFINES and all that - hello the 90ies :)).

But exactly this fact - the lingua franca C - makes it perfeclty “wrappable” into any modern language. Even Apple Swift compiles and links without further ado with C code! And so does Golang (from what I’ve quickly read), and I guess with some effort even Java (UPDATE: of course: with the Java Native Interface) etc., not to forget C# and of course C++.

So you can hide all those gory “SimConnect HANDLE” handlers, the structs, the #DEFINES, the procedural style…

And I don’t quite follow your argument about “limitations”: what “limitations” of the SimConnect API are you referring to specifically? I mean if a given functionality is not implemented in FS2020, you simply can’t use it - regardless of wheter you’re trying to “sneak into the direct network communication” or whether you’re trying to call the (missing) functionality via the SimConnect API (I am thinking of “non-settable simulation variables” here).

Regardless, sounds like a very interesting project and I think especially the “cross-platform nature” is worthwhile! As I am still busy with my own little pet project (Sky Dolly) I cannot help out right now, but I do hope your project finds interest in the community!

1 Like

I thought exactly the same as there are (fairly) regular updates with the SmConnect SDK, which implies there are protocol changes to match.

Then I started looking into the documentation and code available and found the Protocol itself is quite static. In fact, since the first API was developed, there have only been 5 versions of the protocol itself.

I believe the SDK releases are to fix bugs and implement new features, using the same protocol for data transfer.

Even better than that, the API is (apparently - according to the documentation I’ve found) backward compatible, supporting all protocols except the very first one (which I’m led to believe is the FSX Beta protocol and is no longer used anywhere).

This is also corroborated by the SimConnect SDK itself. One of the parameters available when instantiating it is a simple boolean value to inform the SDK you wish to use the FSX-Compatible protocol:

new SimConnect("ClientName", handle, WM_USER_SIMCONNECT, null, **UseFSXcompatibleConnection**)

So with a little luck, it won’t be that big a headache to create classes to emulate the latest Protocol version, which is likely to remain supported for quite a while, allowing plenty of time to update the code whenever a new version of the Protocol is released.

I’m not trying to bypass the API, it’s the SDK itself I’m trying to avoid, giving us more options and the ability to communicate with the API from almost anywhere, by demystifying the network protocol used with the API and allowing us to simplify the communication.

That’s exactly the aim of this exercise, we don’t need handles, pointers, structs, etc, just to send an array of bytes and process the results, hence I’m hoping that removing the reliance on all of that, and concentrating on handling the data stream to/from the API, we might even improve performance (although the SimConnect SDK is pretty good at performance already), reduce overhead and allow us to port code to almost any language, perhaps even scripting languages like JavaScript.

It’s not API limitations I’m trying to overcome, it’s the SDK we currently use to communicate with the API, which as you’ve already identified, has many old-school, dated practices such as reliance on handles, pointers, structs, etc.

As my primary development language is C#, I’m using .NET Core 5.0, which is already ported to Linux, Apple Mac, etc. However, as I intend to publish the code as Open Source, Free to Use, usual disclaimers on suitability, etc anyone will be free to port the code to any other language, to provide native support on their own chosen platform.

This will allow people building haptic feedback, simulation rigs, etc, to embed communication within their PIC devices, removing any reliance on requiring a PC to interact with the API and handle the communication for them. Imagine an Arduino or Raspberry Pi that can be used 100% stand-alone to update dials etc, with no PC needed via the USB port, etc.

In an effort to assist with this, I’m trying to ensure all methods, etc are documented within the code, helping others to fully understand its purpose, parameters and return value(s).

2 Likes

You are absolutely right: whenever Asobo would change the network protocol in a “binary incompatible way” that would also mean that all existing 3rd party addons would become incompatible (and would have been relinked with the latest SimConnect.dll, as described above). But that’s definitively something that Asobo (Microsoft) would like to avoid!

1 Like

You are absolutely right: whenever Asobo would change the network protocol in a “binary incompatible way” that would also mean that all existing 3rd party addons would become incompatible (and would have been relinked with the latest SimConnect.dll, as described above). But that’s definitively something that Asobo (Microsoft) would like to avoid!

Please don’t forget that the “SimConnect DLL” is not the C language API, as there is also a static library. There is also a Managed API, even though that does not mean it is much more modern in its design. This is what is targeted in the CsSimConnect library: provide a modern C# API. At the back, it uses the C static library, although I intend to work with Dragonlaird to get his backend usable with my API as well. Then you can use modern .Net Core on multiple platforms.

As for the risk that Microsoft, Asobo, or Lockheed Martin make changes to SimConnect, that is actually already a problem: Prepar3D has way more calls available than MSFS. Actually, the MSFS API is a lot closer to the original FSX one. All the more reason to put a layer in between and protect us, developers, from the backends.

I have tried to contact Microsoft several times, but they have so far not responded. Asobo did, actually, but only by assuring me I needed to get in touch with the Microsoft team.

Yes, you are right. The API is embedded within MSFS itself. Many people get confused between an API and an SDK as they perform a similar, yet distinctly different function.

  • An SDK is the component/library you embed within your solution, to interact with the destination application.
  • An API is an endpoint embedded within the target application (e.g. MSFS), often used by an SDK, to receive requests and/or provide a response to requests.

In this instance, the SDK (either Managed or Unmanaged) is in the form of a DLL, which is not uncommon, as it a re-usable library of methods, properties, and events for the solution to use.

My aim is to bypass the SDK completely, allowing us to demystify the Protocol (communication process) used by the SDK/API, and thereby permit us to communicate directly with the API, not to provide a replacement to the existing SDK.

There’s a managed SDK (e.g. C# etc) and an unmanaged SDK (e.g. C, CPP, etc.). The API is still the endpoint within MSFS and is a network listener, allowing direct communication with MSFS itself. The SDK simply parses your requests and passes them to the API and provides the response returned by the API back to your solution.

So far, I’ve figured out how to initiate the handshake between the client and MSFS. Essentially, this “registers” the client with MSFS, so that MSFS remembers the client and which values it has requested, etc.

I’m in the process of figuring out how to request a SimVar and parse API responses, I think I’ve figured out an easier way to build requests and parse responses, time will tell.

Folks, there is a substantial difference between what you call an “SDK” and what I believe most people would say an “SDK” is (the same goes somewhat for the well-known term “API”).

Of course you can call your own work whatever you’d like, but I’d suggest you stick to the “generally used terms and definitions/meanings” if you intent to implement a new API.

So let me quickly write down the “generally accecpted definitions” of SDK and API:

  • SDK simply stands for “software development kit”: I couldn’t spell it out better than Wikipedia does, quote: " A software development kit (SDK) is a collection of software development tools in one installable package."

So it is not a single DLL (a DLL is simply an implementation detail, if you wish), but really the entire set of what the vendor provides to 3rd parties to create add-ons etc.

Ever wondered why Microsoft is calling the installation folder of the SDK “MSFS SDK”? Exactly! :wink: Because that’s the folder where the entire software development tools are contained: that includes also applications (tools) for creating scenery (if MS provided such tools, never have checked all what’s in the SKD myself ;)), possibly also compilers, shaders, paint kits… that’s what people usually refer to as “SDK” - as “software development kit”.

  • An API is basically the “set of methods, functions, types and (global) variables” that allows an application to interact with another application (or “system”). In fact, an API specification can be totally separated from an actual implementation - think Open API: “API first” development etc. Or as Wikipedia puts it: " In computing, an application programming interface (API) is an interface that defines interactions between multiple software applications or mixed hardware-software intermediaries.[1] It defines the kinds of calls or requests that can be made, how to make them, the data formats that should be used, the conventions to follow, etc"

So when we talk about the “SimConnect API” we are of course not talking about the actual implementation (which is contained essentially in the client SimConnect.dll - or the static variant thereof, doesn’t matter). We are of course talking about this:

https://docs.flightsimulator.com/html/index.htm#t=Programming_Tools%2FSimConnect%2FSimConnect_API_Reference.htm

See? It even has “API” in its name :wink: So this constitutes the functions and types with which the client (“your app”) interacts with the server (here: FS 2020).

The fact that a “managed API” (C#) exists - and many “wrappers” for other languages - doesn’t impact the generality of what I just said about “API”, we are still essentially talking about functions like SimConnect_Open and friends. Or in other words: the C# SimConnect API is an “adaption” of the “API specified for the C language” (the SimConnect.h essentially), to make it (better) suitable for C# programming conventions (and to make it callable in the first place, of course).

Now when you say “My aim is to bypass the SDK completely, allowing us to demystify the Protocol (communication process) used by the SDK/API, and thereby permit us to communicate directly with the API” that is actually wrong (in terms of how people generally use the term “API”): yes, you are communicating with some API (*), but that is the internal, undocumented (as you have already figured out ;)) server-side API which is not public (and hence not meant to be used directly).

(*) You could say "everything is an API then! Even a simple function that returns “Hello, World!”. And in fact you would be correct :wink: But when we talk about an SDK and it’s API we are mainly interested in the “public and documented API that developers are supposed to use”.

The way a 3rd party app is supposed to communicate with FS 2020 is via the official, public and documented API, as defined here:

https://docs.flightsimulator.com/html/index.htm#t=Programming_Tools%2FSimConnect%2FSimConnect_API_Reference.htm

Again, we are simply talking “naming / definition differences” here and I don’t mind if you keep / use your own definitions and terms. But it did create some confusion for me :wink:

2 Likes

(Darn, too late)
Sorry, but I also beg to differ in this case. There are two “Published APIs”: One is specified in a C include file “SimConnect.h” and implemented using the SimConnect static C library and the SimConnect C DLL, the other is specified and implemented in a Managed DLL. They have been described in the SDK documentation. An SDK provides tools and docs to allow you to develop applications for a given Platform or against a given API.

The SimConnect messaging protocol is internal and not publicly specified. All simulators provide several APIs based on that protocol, both via IPC and TCP/IP.

1 Like

We were obviously on the same wavelength this morning…

Anyway, enough about terminology, time to get back to the actual topic, the SDK/API Protocol.

As I mentioned in a post above, I’ve documented the initial connection request and I’m most of the way through the initial response (parsing it, so it’s readable, I’ve figured out what it contains).

Also had a quick look at the format of requesting SimVars, both initially and also requesting updates, it looks straightforward.

Once I’ve got all of the above documented, I’ll open the GitHub repo, as that should provide the basics needed to start communicating directly with the API for most.

1 Like

Exactly :slight_smile: And I just noticed that I posted my previous reply into the wrong thread, so here again (copy & paste):

To be clear: I am still intrigued by your proposal to implement a “platform-independent API” (API in the terms given in my previous post, to be clear, he he ;))

Or specifically: replace this:

https://docs.flightsimulator.com/html/index.htm#t=Programming_Tools%2FSimConnect%2FSimConnect_API_Reference.htm

with “something nicer”.

For instance the current C API (-> SimConnect.h header essentially) has many “Window-isms” and dependencies, for instances types like HANDLE and DWORD which could easily be replaced with platform-agnostic types.

But that is the easy, “mechanical work” part. The tricky part is of course: “what is a good API”, also in terms of “modern language concepts”? I’ll leave this up to you to figure out for now :wink:

Going one step further with this one could even try to specify the API really “platform / language” independent, e.g. with an API specification language like the mentioned OPEN API.

(In short what OPEN API does: it is a YAML (or JSON) based specification of your API: the functions (methods), types and their interrelationship. Given this specification you can then create the (boilerplate) code (“types and method signatures” - the later are of course still to be implemented with the actual “business logic”) for your desired language (Java, C#, JavaScript, golang, … theoretically any programming language, for as long as a generator for it exists)

But given the fact that the “main challenge” of your work will be to "decipher the internal (communication) API I would suggest to keep it simple and stick to your favourite language (which is C#, as I understand ;)) and define your new public API “in code” (in C#).

Now designing your API is also tricky for another aspect: you need to “convince” people to use it! So existing code should be “fairly straight forward” to port to your new API - so records and functions should be “similar” somehow; at least a “mental mapping” should exist. The simplest case would be: "if your code calls “HANDLE handle = SimConnect_Open(…)”, now call “simConnect.connectWithServer(…)” (or whatever).

Because if the porting effort is too big then people might shy away from using it. Or you focus on “new applications” with no porting effort!

In any case, the gain would be a “platform-independent, nicely designed API”, dealing with modern concepts with “asynchronous calls” (-> futures, promises, … all those concepts). It’s a huge challenge - both deciphering the internal API, but especially come with a good, well-designed attractive API.

There is one caveat though that just came to my mind: the fact that Asobo actually deprecated the “network functionality”:

SDK Documentation (flightsimulator.com)

Quote: " IMPORTANT! This section relates to a legacy process intended for use with Microsoft Flight Simulator X. It is still valid for Microsoft Flight Simulator but may be changed or even removed at some time in the future and should only be used for support of legacy add-ons."

(We’ve had this discussion already in another thread ;))

Which could imply that Asobo would - at some point - completely re-implement their “internal API / communication”.

Or - in the best case - it could simply mean that the *.CFG configuration files are dropped (and replaced by a “proper API”, to be able to programmatically set the network configuration).

But that could be years away

@Steeler2340 I’m not trying to develop an API, I’m trying to understand the underlying communications protocol used by the existing API/SDK, so we can potentially take the SDK out of the loop entirely, and talk directly to the API itself.

There is a likely benefit to developing an Open API solution but I’m trying to walk before we run. Once I’ve documented the protocol, all options like this become open to us.

The annoying thing is, the API comms are actually quite simple, why Asobo/MS couldn’t release it, is beyond me. It literally is just a network endpoint that accepts and returns a binary data stream.

The potential benefits of doing so are not to be understated, people can write and implement their own libraries for communicating with the API, in any language, remove intermediate code that’s not required, improve performance, minimise required code and so much more.

Because, again, that is not the public API. One is not supposed to “mess around with the internal protocol (API)”.

That’s why it is not documented - because it does not need to be. A developer is supposed to develop against the public API, which is documented here (and “variants” thereof, like the corresponding C# API):

SDK Documentation (flightsimulator.com)

So unless you want to come up with your own, “nice and clean” and platform-independent API (replacing the SimConnect API as described in the link) I don’t see the value in deciphering the internals - why would I want to write my application against the “raw, undocumented internal API”) and dealing with “structured streams of bytes” (apart from the “platform independent” part)?

That’s feels like e.g. deciphering the internal API of some GPU driver and stripping away the public API (OpenGL, Vulkan, DirectX and Metal) and say: “here are the GPU internals! Now you can write platform-independent GPU code!” :wink:

2 Likes