Has anyone gotten SimConnect to repeatedly get data?

[Steeler2340]

@MSGamerTag01

Steeler there’s a crucial additional step which does make it work the way intended:

The RequestDataOnSimObject(…) call includes a reference to a callback function as a parameter (unlike the RequestDataOnSimObjectType call).

Well, that’s exactly what I would expect, but unless I am totally blind or stupid (or both) we seem to talk about different variants of SimConnect_RequestDataOnSimObject!

The one I am referring to:

SimObject Functions

does not have such a callback parameter! And again, all the provides examples seem to suggest that you need to poll for messages, with SimConnect_CallDispatch.

UPDATE: And just to be clear: the SimConnect.h header file from the FS 2020 SDK (the previous iteration - haven’t downloaded the SDK from the “December update”, in case that’s different anyway) does not have such a “callback parameter” either: so it seems to be consistent with the “Prepar3D” API docu (even though I understand the current FS 2020 SimConnect.dll supports way less functionality than the Prepar3D API).

Examples given:

• Sample: ClientEvent.cpp
• Sample: TaggedData.cpp
• Sample: RequestData.cpp

All those examples have a “while(true)”-kind of loop which keeps polling for messages in 1 second (sleep(1)) intervals.

So unless you can point us to a concrete example or API documentation I just don’t see how to avoid this polling code pattern (except this funky “DLL start” voodoo magic, which I have not tried yet and which may only work with the Prepar3D specific implementation anyway…).

[Steeler2340]

@FerretWitASpork

Maybe consider an easier language to start with. Python is very beginner friendly. This is a SimConnect library for Python

In fact, I looked at several “wrapper libraries” how they implemented their callbacks, and just now I also looked at this Python library (over which I already stumbled previously as well):

And lo and behold:

Python-SimConnect/SimConnect/SimConnect.py at master · odwdinc/Python-SimConnect · GitHub

 def _run(self):
	while self.quit == 0:
		self.dll.CallDispatch(self.hSimConnect, self.my_dispatch_proc_rd, None)
		time.sleep(.002)

A while loop with a sleep call - bang! That’s polling.

And to clarify: I am totally not trying to “defend” this pattern - I am merely trying to find out how else it could be done (“without polling”), that is, to get the SimConnect.dll logic have my callback called without me having to poll for messages (which kind of defeats the idea of a callback in the first place).

[MSGamerTag01]

Steeler APOLOGIES I totally misread the RequestDataOnSimObject() docs - you are of course completely correct and my description was mistaken.

[Steeler2340]

No worries

Well, there might be another way to react to messages received by SimConnect.dll, and that is the Win32 event messaging:

From the docs:

General Functions

There are 3 parameters related to Win32 event handling (which requires a hWnd instance):

hWnd
[in] Handle to a Windows object. Set this to NULL if the handle is not being used.
UserEventWin32
[in] Code number that the client can specify. Set this to 0 if it is not being used.
hEventHandle
[in] A Windows Event handle. A client can be written to respond to Windows Events, rather than use a polling and callback system, which can be a more efficient process if the client does not have to respond very frequently to changes in data in Prepar3D.

In fact, the API documentation seems to reiterate on the point that the alternative is indeed polling.

So instead of providing a callback function and polling for messages with “dispatch” one can apparently “listen” for (user defined) events injected into the Win32 event queue (which otherwise provides access to “window events” such as mouse movements, repaint events etc.).

But as the documentation also indicates this may only be preferrable if few events are expected. But if one is going to permanently track the aircraft’s position (possibly every “frame”) then the polling approach indeed seems to be the way to go…

I have yet to try this “DLL start” approach (which apparently “automatically” calls the callback, without the need to repeatedly call “dispatch”).

[poopyBeard86]

Sounds intresting. Please let us know if you manage to get this Win32 event approach working.

[Vibstronium]

@MSGamerTag01

@Vibstronium any chance of you posting your complete working RequestDataOnSimObject example? Examples are almost impossible to find so yours might be “it”.

Sure thing, I’m updating my current work skeleton code now.

[poopyBeard86]

I wrote a little example to show how i use RequestDataOnSimObject()
If somebody has a better way (without calling dispatch all the time) to do it please let me know…
I’m curious about Vibstroniums example.

This code prints the throttle lever percentage to the console only when the values are changed:
(boost is just used to format the prints. if you don’t have boost installed just remove the formatting of the prints.)

andrezwicky/ThrottleWatcherExample

[Vibstronium]

brion/WikiWindowSeat

Currently very bare bones just connects to the sim and queries alt/lat/lon of the user and reports it back. It’s working with the single request on a 1-second period, done with C#.

[Vibstronium]

@AndreZwicky

If somebody has a better way (without calling dispatch all the time) to do it please let me know…

It looks like you have a single-threaded app with no event loop, so you’re just spin-looping on SimConnect_CallDispatch even when there’s nothing to handle; this will ramp up your CPU usage to max out one core.

SimConnect uses a win32 window message event as the inter-process trigger to tell you there’s new messages to read and dispatch to your callbacks; thus it piggy-backs on the Windows event loop and your app can stay idle while waiting for new data to arrive.

You should be able to create a single hidden window (bare win32 calls should be fine if you don’t want to play with modern frameworks) and pass its window handle and a preferred window message ID from the user area (the examples all use 0x0402) to SimConnect_Open. Then in your window procedure, when you receive that window message call SimConnect_CallDispatch until it returns 0.

[poopyBeard86]

Thanks for the pointers. I will look into those windows event loops.

[MSGamerTag01]

Thanks Vibstronium awesome stuff…

[Steeler2340]

@Vibstronium

Then in your window procedure, when you receive that window message call SimConnect_CallDispatch until it returns 0

Are you sure about that? Or asked differently: don’t you get a user “window event” per message? Because according to the (Prepar3D) API docs:

General Functions

The function returns an HRESULT. Possible values include, but are not limited to, those in the following table.

Return value	Description
S_OK	The function succeeded.
E_FAIL	The function failed.

And given that:

#define S_OK ((HRESULT)0x00000000)

so a return value of 0 simply means that the “dispatching” has succeeded (and not “no more messages to process”).

Also in your example - thanks for that - your comment about the C++ case (// In C++ you’d call SimConnect_Dispatch() in a loop here) is asking for the same question: “How long would you loop? What would the loop termination criterium be?”

Anyway, what I am actually trying to get at is the following: assuming I would setup the data retrieval for some given object, say the user’s aircraft, and asked the data to be updated (sent to the client) for every frame, as in:

res = ::SimConnect_RequestDataOnSimObject(d->m_simConnectHandler, AircraftPositionRequest, AircraftPositionDefinition, SIMCONNECT_OBJECT_ID_USER, SIMCONNECT_PERIOD_SIM_FRAME);

So messages would be sent for every “simulation frame”. And let’s further assume that I would (want to) call the SimConnect_CallDispatch only every second, or for the sake of argument, even only every 10 seconds.

So the questions:

• Are the received messages buffered (at least for a certain amount of seconds / buffer size) in SimConnect.dll?

• If so, is my callback called for every buffered message (since the last call to SimConnect_CallDispatch) when I call SimConnect_CallDispatch only once (“once” as in “once every ten seconds”)?

• Or do I have to call SimConnect_CallDispatch until “no more messages are there to process”? → But that then brings up the question again: how to detect that there are no more questions (because as already mentioned above: an S_OK = 0 return value simply means that “dispatching was okay”.

UPDATE: Oh, and a follow up question to “receive messages every frame, but dispatch only every 10 seconds”: I just set a breakpoint in my dispatch function and inspected the received data structure (for the reply to the above SimConnect_RequestDataOnSimObject), specifically the SIMCONNECT_RECV * pData structure, but I did not find any sort of “timestamp” information about when the message arrived. Is there a way to check / get such a timestamp, especially if I am processing a message only 9 seconds later?

Because if not, then my intention of polling for messages only every 10 seconds (or 1 second, to be more realistic) is moot anyway if I cannot assign every assigned message (its data) a proper timestamp… (use case: to record the user’s aircraft position).

Thanks!

[Steeler2340]

@Vibstronium

It looks like you have a single-threaded app with no event loop, so you’re just spin-looping on SimConnect_CallDispatch even when there’s nothing to handle; this will ramp up your CPU usage to max out one core.

Yes, I absolutely agree with this assessment:

while (quit==0) {
    SimConnect_CallDispatch(hSimConnect, MyDispatchProcRD, NULL);
}

This loop is absolutely going to burn your CPU At least but some „sleep“ statement into this loop, e.g. poll at 60 Hertz or whatever.

Also checking state changes of the thrust levers is probably a good example where you should indeed use the „win32 event loop“ („user events“) approach, as thrust levers are going to be changed relatively infrequently. And (only) processing messages once they have arrived would be more efficient, especially if you‘re only interested in state changes.

[Steeler2340]

@Steeler2340

while (quit==0) {
    SimConnect_CallDispatch(hSimConnect, MyDispatchProcRD, NULL);
}

This loop is absolutely going to burn your CPU

Ah okay, now I understand where this seems to come from:

SimConnect | Microsoft Learn

In the example given for SimConnect_CallDispatch there is the exact same „burn your CPU“-loop. Bad, Microsoft, bad, bad, bad

At least the Prepar3D specific documentation (which seems to be derived from the above documentation almost verbatim) added a „Sleep(1)“ statement into those polling loops

UPDATE: Nope:

General Functions

The exact same code example is given here. But at least the (most) working examples have a Sleep call…

[Vibstronium]

@Steeler2340

So messages would be sent for every “simulation frame”. And let’s further assume that I would (want to) call the SimConnect_CallDispatch only every second,

No, you would want to call SimConnect_CallDispatch every time you get the window message, which will be at least once per simulation frame because you requested to get data every simulation frame.

[Update: to clarify – there is likely a buffer (probably a ring buffer with a maximum size) holding the incoming messages, but I don’t think it’s meant to queue things up arbitrarily long; you’re meant to dispatch the callbacks and handle incoming messages as fast as you can.

Also, unless you specifically asked for the current time in them the messages will not contain a timestamp, as far as I know. You should probably dispatch messages immediately, assign a suitable timestamp, and then queue them on your own if you only want to process them every 10 seconds or whatever.]

[Dragonlaird]

SimConnect uses Windows Messaging to notify your application whenever it has a new value for any of your requested SimVariables (aka SimVars).

When this event is overridden within your code, simply identify if the message came from SimConnect then call the method to instruct SimConnect to call your appropriate method that you registered.

The hardest part is writing the message handler for SimConnect to call initially, as this would typically be built in to Windows objects such as Forms, if you want to create an application designed to run without a Form, you need to create a MessagePumpHandler.

I can provide some sample code in C# on how to achieve this, if it’s required.

This is how you use SimConnect without polling for updates.

[Steeler2340]

@Vibstronium

No, you would want to call SimConnect_CallDispatch every time you get the window message, which will be at least once per simulation frame because you requested to get data every simulation frame.

[Update: to clarify – there is likely a buffer (probably a ring buffer with a maximum size) holding the incoming messages, but I don’t think it’s meant to queue things up arbitrarily long; you’re meant to dispatch the callbacks and handle incoming messages as fast as you can.

Yes, I agree. After thinking about it a bit longer, and given the fact that there is no timestamp in the original messages I also believe that the design of the SimConnect API is that you “pick up the message as soon as possible (after it has arrived)”.

This is actually supported by the API documentation (which is basically the same as the corresponding - and still available - Prepar3D API documentation):

SimConnect | Microsoft Learn (*)

For function SimConnect_RequestClientData:

SimConnect

RequestID
[in] Specifies the ID of the client-defined request. This is used later by the client to identify which data has been received. This value should be unique for each request, re-using a RequestID will overwrite any previous request using the same ID.

As the RequestID won’t change for requests done with a given period (e.g. “per frame” etc.) I strongly assume (without having verified it in practise) that any previously received result message will simply be overwritten - as per the documentation - with any newly arriving response.

But yes, I also assume that this response will be stored in a buffer; large enough to hold exactly one response record (as defined by SimConnect_AddToDataDefinition), until either “dispatched” or the next response for the same RequestID arrives.

Bottom line: so yes, one either needs to “poll” with the same (or higher) frequency as the messages arrive (are expected to arrive), as per the documentation of

SimConnect | Microsoft Learn

→ “to call this function sufficiently frequently”

or get informed by the Win32 event queue, with a “user event” whenever a message (of any SimConnect type) has arrived (and then call “dispatch”).

The Win32 event queue however is only favourable if events happen rarely (relatively speaking). If events are expected to change frequently (such as the aircraft’s position) then “polling” is more performant (according to the API docs).

(*) This is the original (I assume) Microsoft SimConnect API documentation, included in a “flight simulation development kit” called ESP (whatever that stands for). This product has been discontinued, and Microsoft is most likely exluding those pages from any search engine robot: even with a search like “SimConnect site:docs.microsoft.com” (and more precise keywords) you won’t find this resource anymore (neither with Bing nor Google ;)).

[Steeler2340]

@Steeler2340

But here comes the interesting part:

„ However, if the project involves developing a library (DLL) rather than an application (EXE) then only one call to this function is necessary. This call will store the name of the callback in a cache, and whenever a packet is sent to the client, the callback function will be run“

And I think if have now a better understanding of this “DLL voodoo magic” as well!

Again from the API docs:

SimConnect

“The recommended method of writing an add-on is to build it out-of-process, as an application (an .exe file) rather than in-process, as a library (a .dll file). This is because out-of-process applications provide more stability, if they crash they will typically will not crash ESP, and are easier to build, test and debug.”

Or in other words: it is (or was) apparently possible to extend the flight simulator platform (Flight Simulator X, Prepar3D, FlightSimulator 2004 etc. - not sure about FS 2020) functionality with a DLL (“plugin”) which then runs in the same process space as the flight simulator itself!

And now it all makes sense: calling “connect” from within DLLStart() is expected to always work, because at the time the plugin is instantiated (the DLL is dynamically loaded) the server part is already up and running!

And since the entire plugin logic is running in the same process space like the flight simulator I guess that SimConnect_CallDispatch - when called from that DLLStart() function - will register the callback with the flight simulator itself. Hence (I assume) it is the flight simulator (“server”) logic itself which calls our callback, just after the server has sent a new message.

Or put differently: only if we extend the flight simulator platform itself with a DLL (“plugin”) is this “DLLStart() callback registration” pattern possible. But not if we are talking about DLLs which are linked to our own, separate executable (which is the case for me).

[Steeler2340]

@Dragonlaird

SimConnect uses Windows Messaging to notify your application whenever it has a new value for any of your requested SimVariables (aka SimVars).
…
I can provide some sample code in C# on how to achieve this, if it’s required.

Yes, thank you - we have already established above (Has anyone gotten SimConnect to repeatedly get data? - #24 by Steeler2340) the fact that with the Win32 event queue and user events you can get informed about new SimConnect messages, without polling. And @Vibstronium has also already posted a working example (Has anyone gotten SimConnect to repeatedly get data? - #28 by Vibstronium) using the Win32 event queue above

But again: using the Win32 event queue is only more efficient in case infrequent (rare) events are to be handled (the event queue poses a considerable overhead, too):

SimConnect | Microsoft Learn

“A Windows Event handle. A client can be written to respond to Windows Events, rather than use a polling and callback system, which can be a more efficient process if the client does not have to respond very frequently to changes in data in ESP.” [emphasis added]

[Dragonlaird]

Thanks Steeler,

Apologies if I was covering old ground, I confess I hadn’t read the full thread.

With regards to the Windows Events being inefficient when frequent events are to be handled, I would have to slightly disagree.

I’m in the process of developing a client/server scenario for remote desktops, which requires many variable requests to be handled (each displayed instrument uses at least 1, usually several, SimVars), which are requested once but have many values being supplied by SimConnect.

It all depends on how you handle the WndProc events being raised.

If you are processing each raised event within the same thread or handling each request from start to end, then yes, it is inefficient and costly.

If you’re filtering the events and initiate the handler within its own thread, by invocation, then it’s very lightweight.

When debugging my code, I am monitoring the entire client/server environment, with many instruments being displayed and updated. The server component is by far the least costly in terms of resources and works extremely well.

You would only ever need to handle each event within the WndProc method if it was absolutely time-critical, but given the overall speed of how things respond in an aircraft, I cannot think of a single scenario where you must know of (and respond to) something the exact microsecond it happens - I’m happy for you to disagree.

I intend to publish the Client/Server code soon for anyone to use, once I complete the code changes needed for handling remote images (e.g. a GPS instrument). So my offer to share a snippet of code above is a little premature but was always planned.

The other benefit of using the Client/Server scenario is that it removes the obfuscation and complexities of the annoying nuances when dealing with SimConnect directly.

The code I’m developing also allows access for remote applications by exposing an API to request SimVar values and can co-exist and interact with applications developed in virtually any language, accessible from almost any PC on your network.

It also has many other features built-in, such as auto-notification when FS is running or stopped, caching the latest values for each variable requested, remembering which client has requested which variables, the last time each value was updated, the last time each client was updated, auto-resubmission of previous requests when FS restarts and many more.

I’m hopeful the code will, in some form, be helpful to any future developers wanting to use SimConnect, and perhaps provide a simpler interface for those wishing to avoid dealing with SimConnect entirely.

Dragonlaird