C++ continous data transmission with and without SysEvent subscription

Thanks Andre - merry xmas! I’m looking for any sample simconnect program that uses the RequestDataOnSimObject as intended i.e. you do NOT have a client-side timer and loop there polling for data from the sim. Instead you ‘subscribe’ for the data and the sim CALLS your callback.

Examples for this are incredibly rare, with developers naively copying e.g. the Request Data sample or the SimvarsWatcher sample which ‘poll’ for data which only works for the most basic low-rate apps.

I am curious about this, too. In fact, polling is an „accepted“ way of querying sim vars, as it seems. As you say, all tutorial examples I have found so far - including actual applications on e.g. github.com - use either a timer or a „polling loop“ (even as mercilessly to the CPU as „while (true) loop… sleep(1) end loop“)

Interestingly though:

„ It is important to call this function sufficiently frequently that the queue of information received from the server is processed (typically it is coded within a while loop that terminates when the application is exited)“

So this seems to confirm that polling is the way to go. But…

„ However, if the project involves developing a library (DLL) rather than an application (EXE) then only one call to this function is necessary. This call will store the name of the callback in a cache, and whenever a packet is sent to the client, the callback function will be run“.

An example DLL setup with int __stdcall DLLStart(void) and void __stdcall DLLStop(void) is given.

So from what I understand from this, if you implement your SimConnect logic in a DLL:

  • You connect to the flight simulator when the DLL is loaded (started), and…
  • … call SimConnect_CallDispatch in that same „start“ function, but only once.
  • Due to some black magic the actual SimConnect.dll logic then „remembers“ (caches) your callback and - according to the documentation - calls it whenever messages (sim vars, possibly also sim events…) arrive

Now why one cannot register the callback by other means such that it is called whenever messages arrive (except this DLL voodoo magic) is beyond me. And quite possibly this may be specific to the Prepar3D implementation!

And there is an obvious disadvantage, too: the flight simulator already needs to be up and running in order to be able to connect when the DLL is loaded.

(Not sure whether the same mechanism works when dynamically loading/unloading the DLL (as a „plugin“), which would allow to „reconnect“ dynamically at runtime of the client application).

I haven‘t tried this DLL approach just yet (I am only like three days into my own experiments ;)), but would be interested in any findings, too. Especially whether there is not another way to register the callback, such that we can avoid calling „dispatch“ in a polling way.

The people (person?) that developed Simconnect back in the early 2000’s absolutely knew what they were doing. Gaming code has been ‘event driven’ for ages. It’s just mainstream programmers have never really seen the async style of programming and so get it wrong over and over again (so yes, there is a huge amount of polling code out there, but it’s pretty much all wrong). Back in 2000 the only mainstream method available for async programming was the ‘callback’, so simconnect supports this. More generalised capabilties have been added to languages more recently (async/await) but the callback method works fine (and is the central style of programming in javascript on mouseclicks, keystrokes, http data return, etc).

Simconnect has ‘Simconnect_CallDispatch()’ where you can define a completely general callback which the sim will call once per its internal update cycle. Inside your CallDispatch callback you basically have a case statement where you work out what events have occurred since you last got called, including new simvar data available for previous RequestData subscriptions.

Simconnect also has RequestDataOnSimObject(… …). This is similarly ‘event driven’ but this time specifically giving your callback the simvar values on each update cycle (which you can set as per frame, per second, …).

Neither Simconnect_CallDispatch() or RequestDataOnSimObject() should be considered ‘polling’, as both result in your code being called by the sim, not the other way round.

An example of why polling is problematic: at the moment I am looking at data through simconnect with a program that polls for the data 10 times/second. It is clear MSFS is updating the underlying simvars I’m using 18 times/second (e.g. if I ask for “E:ABSOLUTE TIME, seconds” it always changes on a 0.0556 second boundary). The difference between the sim update cycle and the polling frequency effecfively represents a ‘jitter’ in the values that has to be programmed around.

1 Like

My understanding is that a DLL runs in the same process as the simulator, so your callbacks can be called directly. A separate executable, however, has to have data routed through some sort of shared memory segment, and a win32 user-defined message event is used to wake up your process and tell it it’s time to go through the memory segment dispatching incoming events.

So calling SimConnect_CallDispatch is not “polling” in the sense that it doesn’t make any inter-process communication calls; it most likely just reads event records from a ring buffer.

But it does have to be called in a C/C++ executable, or its C# managed equivalent SimConnect.ReceiveMessage, to dispatch the callbacks.

I think maybe the CallDispatch call blocks inside simconnect until the data is ready, without blocking the main sim. Not sure.

Well, unless you know about some other way (which is not made obvious in the API docs - at least the ones from Prepar3D) the “dispatch” is actually supposed to be called by the client (“my code”).

So instead of polling for actual data with e.g. RequestDataOnSimObjectByType (which we need to call each time we are interested in the corresponding data) we are now polling for messages (which I assume we might do less frequently than with RequestDataOnSimObjectByType, because it is my assumption that “dispatch” will call my callback for all messages which have been received since the last dispatch call).

But again, I am quoting the Prepar3D API documentation verbatim, for SimConnect_CallDispatch: " It is important to call this function sufficiently frequently that the queue of information received from the server is processed (typically it is coded within a while loop that terminates when the application is exited)"

https://www.prepar3d.com/SDKv5/sdk/simconnect_api/references/general_functions.html#SimConnect_CallDispatch

And that in my book is the very definition of polling (this time for messages).

Again, if you can show us (with a workgin example in the best case, or a pointer to the corresponding API) how to “register” our callback such that it gets called “from SimConnect.dll” automatically then we’re all ears.

As I already wrote there seems to be a way to do this, with the mentioned “DLL black magic voodoo” (as mentioned in the above API doc link)

No need to explain this again :wink: (Btw I am a professional software engineer since 20+ years - but yes, totally new (as in “a couple of hours”) to the SimConnect API ;))

Hold on: do not confuse the SimConnect.dll with the DLL the API documentation refers to: when the API doc is mentioning a DLL library (“However, if the project involves developing a library (DLL) rather than an application (EXE) then only one call to this function is necessary.”) it is referring to your DLL that you are developing (and which links against the SimConnect.dll).

How else would you modify the int __stdcall DLLStart(void) function of an existing DLL? Well, you can’t (unless you have the code and recompile it).

I assume there are actually two instances of SimConnect.dll: one which acts as server in the FS 2020 executable (or the “server” part is fully implemented in the FS 2020 code itself, don’t know), and the one which is linked against your client.

The one linked against your client (be it an executable or library) could at least theoretically run in the same address space (depening on compiler flags used for SimConnect.dll). But I don’t know whether that is the case with the given SimConnect.dll.

And that is polling: it does not matter where the data comes from (whether from a local file, some shared memory segment or even via TCP/IP or UDP…), the fact that you’re calling a function repeatedly in some given interval (without knowing beforehand whether data is available) is called polling.

So that’s probably simply it: we have to keep calling (“polling”) that “dispatch” functionality, and that’s indeed the intended behaviour.

So that’s probably simply it: we have to keep calling (“polling”) that “dispatch” functionality, and that’s indeed the intended behaviour.

yes, i think there is no way around calling dispatch often. That’s jkust the way it was designed back then…

Should I upload an example somewhere on how to use SimConnect_RequestDataOnSimObject()

edit: i have the feeling you don’t need that anymore and now way more about programming than I do anyways

If you think your example is ready for “prime time”, then by all means :wink:

Btw this discussion is more or less discussing the exact same topic as here:

Has anyone gotten SimConnect to repeatedly get data?

thanks guys - I’m please we’re having this discussion because at the very least we’re agreed on the limitations of the client timer + periodic RequestDataOnSimObjectType calls. For me the ideal would be to have a SimvarWatcher example that used RequestDataOnSimObject (not RequestDataOnSimObjectType, and not with SIMCONNECT_PERIOD_ONCE).

Definitely not wanting to drag this out, but my guess with the CallDispatch method is a bit more nuanced, that it blocks in the sim and returns on the sim update cycle (so it is the equivalent of long polling, which I accept is polling but not quite as independently). That could easily be wrong, so maybe it is indeed just plain polling… (if the ‘sleep()’ call in the while loop is compulsory, then my theory is definitely busted).

i’ll make one…

ill be back

1 Like

updating 14GB first lol

edit: okay i have an example ready, i really just need to wait until MFS updated so i can test it…

1 Like

So i wrote a little example to show how i use RequestDataOnSimObject()

If somebody has a better way (without calling dispatch all the time) to do it please let me know…

This code prints the throttle lever percentage to the console only when the values are changed:

Correct, that’s what I was talking about following what I thought you were talking about. If you weren’t talking about that, then discard my comments.

I assume there are actually two instances of SimConnect.dll: one which acts as server in the FS 2020 executable (or the “server” part is fully implemented in the FS 2020 code itself, don’t know), and the one which is linked against your client.

I don’t think that’s right? SimConnect.dll implements client functions, and doesn’t appear to contain any server functionality. I don’t think SimConnect.dll in any version does server stuff; I think the MSFS 2020 code itself contains the server functionality.

The one linked against your client (be it an executable or library) could at least theoretically run in the same address space (depening on compiler flags used for SimConnect.dll). But I don’t know whether that is the case with the given SimConnect.dll.

Perhaps you’re not familiar with the concept of shared memory segments. Maybe it has a different name in Windows? I mostly program for web services that run on Unix, so. :slight_smile: It’s possible for two processes to both map the same bit of memory into their address spaces. This is commonly used for inter-process communication as I have described.

And that is polling: it does not matter where the data comes from (whether from a local file, some shared memory segment or even via TCP/IP or UDP…), the fact that you’re calling a function repeatedly in some given interval (without knowing beforehand whether data is available) is called polling.

That’s simply untrue. Is calling your callback “polling”? No. It’s not. Because “polling” is when you have to make a system call or other fancy communications method instead of calling a function that reads a structure from local memory address space.

[I would like to affirm again that you DO know data is available because you have been SENT A MESSAGE VIA THE WINDOWS EVENT LOOP. It is ONLY THEN that you call the function which reads from memory in the local address space and doesn’t make any inter-process communication calls, and is in no way polling.]

So that’s probably simply it: we have to keep calling (“polling”) that “dispatch” functionality, and that’s indeed the intended behaviour.

I don’t believe you are using the term “polling” correctly, sorry.

1 Like

Well, a Unix developer - that explains it then :wink:

I am very well familiar with shared memory (IPC) concepts, but in fact I was indeed referring to something completely different here. On Windows you choose from at least four different “runtime” (C/C++) libraries (by setting the corresponding compiler flag), and the choices are:

  • Single-threaded / Multi-threaded
  • Shared Address Space / Distinct Address space (per DLL / EXE)

I am not sure whether the shared / distinct address space is indeed some Windows-specific feature (it’s been a while since I dealt with those questions ;)), or whether that also applies to shared libraries on Unix, but the point is: IF you have a distinct address space per DLL then you cannot share pointers “across DLL boundaries”. And I thought that when you were saying that “My understanding is that a DLL runs in the same process as the simulator, so your callbacks can be called directly.” that you were referring to that “address space feature”.

But apparently you meant something different (related to IPC).

UPDATE: I am referring to this here:

Potential Errors Passing CRT Objects Across DLL Boundaries | Microsoft Docs

E.g. “However, even when the CRT code is identical, you can’t give memory allocated in one heap to a component that uses a different heap.”

That’s because DLLs may use their own “sealed” copy of the C/C++ runtime, managing their own heap etc. - but with the corresponding compiler flags you can enforce a “common address space acrosss all your DLLs” (such that you can allocate memory in DLL A and reference it - and even deallocate it again - in DLL B).

But reading about this I believe that this mess is really a “Windows specific feature” (I can’t remember ever having to deal with that with *.so libraries on Unix/Mac).

Yes. But that is not to what I was referring to :wink:

And that is polling: it does not matter where the data comes from (whether from a local file, some shared memory segment or even via TCP/IP or UDP…), the fact that you’re calling a function repeatedly in some given interval (without knowing beforehand whether data is available) is called polling.

Well, I kind of disagree with your definition of polling here (but let’s not go down further on that road ;)), but I agree on another point with you…

Agreed! In this case you already know that a message has arrived - as notified by the Win32 event queue (or whatever it’s called - I am more proficient with Qt applications than with native win32 development ;)) and I fully agree that when you then call the “dispatch” function that this is not “polling”!

But again, all the above linked examples do not use the Win32 event queue, but really a “polling” loop (a loop with a sleep) - and that is what I was referring to as “polling”.

I am now a bit confused in which discussion thread (this on or the other) we first talked about the Win32 event loop, but that is indeed a way to avoid the “polling” pattern (someone even gave a C# example for this, but I really believe that was in the other discussion thread ;)).

So that’s probably simply it: we have to keep calling (“polling”) that “dispatch” functionality, and that’s indeed the intended behaviour.

Well, I do use the term “polling” correctly. But we have now come to the conclusion that there is indeed a way to avoid polling, by using the Win32 event loop. But according to the API documenation that may be less efficient when you frequently want to retrieve data:

https://www.prepar3d.com/SDKv4/sdk/simconnect_api/references/general_functions.html#SimConnect_Open

hEventHandle
[in] A Windows Event handle. A client can be written to respond to Windows Events, rather than use a polling and callback system, which can be a more efficient process if the client does not have to respond very frequently to changes in data in Prepar3D.

(At least that is apparently the case for the Prepar3D specific implementation).

1 Like

For the record, this is the “other discussion thread” that I was referring to:

Has anyone gotten SimConnect to repeatedly get data? - #21 by Steeler2340

And it is there where we established that the win32 event queue can indeed be used to get informed about new messages, which are then dispatched aka processed in a non-polling way by calling SimConnect_Dispatch.

:slight_smile:

Ok, we’re on the same page now. :smiley:

1 Like

Despite having this discussed across two discussion threads :wink:

1 Like

Wow - that takes me back - we used to use this approach when developing hardware drivers in Machine Code/Assembly - back then it was referred to as Direct Memory Access (or DMA) and was usually event-driven to notify applications when it was updated.

1 Like

Agreeing with you both here:

  • Polling is when you periodically request an update
  • Event-Driven is when the server/remote component notifies you of a change

Both have their uses and drawbacks:

Polling is often used when frequency of updates is pre-set and only needs to be updated/used every so often.

Event-Driven is better suited when every value change is required to be captured, typically for recording purposes and often for low-volumes of change.

Polling is often used for in-process handling of values, Event-Driven is better suited to out-of-process handling, typically using multiple threads to process values, so as not to impact or collide with subsequent events.

The latter is often harder to code for, as the human brain doesn’t often ‘follow’ the logic involved with asynchonous handling, particilarly with multiple threads. The former is often easier to grasp as it follows a more sequential approach.

3 Likes