Has anyone gotten SimConnect to repeatedly get data?

[Steeler2340]

@Dragonlaird

With regards to the Windows Events being inefficient when frequent events are to be handled, I would have to slightly disagree.

I’m in the process of developing a client/server scenario for remote desktops, which requires many variable requests to be handled (each displayed instrument uses at least 1, usually several, SimVars), which are requested once but have many values being supplied by SimConnect.

It all depends on how you handle the WndProc events being raised.

If you are processing each raised event within the same thread or handling each request from start to end, then yes, it is inefficient and costly.

Full disclaimer: I haven’t done any performance measurements myself so far, let alone actually implemented a “win32 event queue” application (I am still a mere couple of hours in my “proof of concept” application, which will basically record the flight track for later replay, that is at 30 Hz or higher (I have yet to figure out a meaningful sampling rate ;)).

So when I said “innefficient” my statement is mostly based on the official API documentation (which I quoted several times above) - and on common sense.

Because you seem to talk about a different aspect of the message processing: the processing of the message itself!

But that was not my point: my point is that at the time you have received the correspoding user event (via the Win32 event queue) you have already - potentially - wasted performace. Before you actually fetch the actual message (be it in the same or a separate thread). Why? Because the Win32 event queue itself is an “overhead”!

And that overhead only pays off if you expect messages infrequently (relatively speaking), or in other words: if you keep polling at a given rate and only get an actual message in 1 out of 10 cases, then your other 90% of “polling” (SimConnect_CallDispatch) calls are “wasted”. In this case the “event based processing” is better (including the overhead of the event queue).

If on the other hand you “catch” 9 messages in 10 “polls” (which implies that the messages arrive more or less at the same frequency as the “polling frequency”) then “polling” is more performant, because it does not involve the costly Win32 event queue (which might not even be there in the first place, e.g. if one wants to provide a “GUI-less” library without a Win32 event queue - so one would have to create a “virtual form” (invisible) simply to get access to the Windows event queue - or so I understand, I am not a “native win32 developer”, I am more proficient with Qt applications).

Again, how you process the message (in a separate “worker thread” or in the same thread) does not matter at this point, because all we are comparing at this point is “polling” vs “event-based message handling” (but yes, that is definitively a separate discussion which also may impact message handling performance, of course - and the same discussion applies exactly the same way for both “polling” and “event-based processing”).

So in your “client/server” use case the Win32 event queue approach may very well be suited better, because as you say, the client is expected to call the values relatively infrequently (at most up to a couple of times per second, maybe at a 6 Hz frequency or even less?). And as you say, the client may even be asking them “only once” (the fact that a lot of variables are requested at once does not matter that much, performance-wise).

I cannot think of a single scenario where you must know of (and respond to) something the exact microsecond it happens

Well, yes, my “flight recorder” scenario: there the aircraft’s position is expected to change every “simulated frame” So I would expect “polling” to be more performant than the “event-based processing”.

But yes, I still have to figure out a balanced (memory and CPU-wise) sampling frequency, because a) positions can be interpolated (during “playback”) and b) sampling at a 6 Hz rate (for both recording and playback) might be already sufficient, because I think FS 2020 “interpolates” (simulates) the aircraft’s position anyway, so (re-)setting the aircraft’s position 6 times a second (or even less, who knows…) might already be sufficient for a “smooth playback performance”.

And in the end it may as well turn out that an event-based processing may be (better) suited for my use-case scenario as well (I am still getting familiar with the API myself and am glad if things work at all ;)).

Oh and yes, I would be very interested in your “client/server” code as well (I have already looked at e.g. a “Python wrapper”, which seems to cache the received messages up to a user-defined interval (like “for two seconds” etc.). So I assume your server also has to cache the received messages (in a ring buffer etc.), until the client has fetched (“consumed”) them, right?

Btw my code will also be eventually made available on github

[Dragonlaird]

You raise some interesting points Steeler, and yes, you are right that we’re going a bit off-topic but I confess, I am intrigued about your Flight Recorder requiring rapid data responses.

Perhaps I should have given some context when I mentioned “time-critical”.

I have configured my sever to only send updates to a client every 2 seconds (this is defined in a config file, so it’s easily changed) to prevent swamping of the network. The server element simply updates it’s local cached values, then the asynchronous socket server sends the latest/current values (or at least those that have changed) when each client connection timer expires.

On the client-side, this translates to rather ‘jumpy’ animations, where a needle would suddenly move from, say 100 knots to 110 knots.

To overcome this, I developed the client to interpolate the ‘missing’ steps and animate (e.g. move the needle) every X ms (50ms seems a good compromise, but again, this can be configured for every instrument).

I could stream the data in real-time from FS, as the server events are run Out-of-Process and have very little overhead, but this only serves to swamp the network with traffic, and is far too excessive for updating a simple display.

Perhaps you could adopt an interpolation approach for your Flight Recorder?

You know where you were, say 2 seconds ago, you know where you currently are when the latest values are received, you could easily fill in the missing ‘steps’ between Point A and Point B, as it’s unlikely you have performed any drastic manoeuvres within a couple of seconds?

The “context” I referred to is visually provided below:

This image shows how much total the CPU usage in my project is attributed to the Event Handler.

And here’s the context, below is how much of the total Memory and CPU available on the PC is used by the entire project when running:

The yellow bar is the Server, the red bar is the server AND client.

So the event handler is using 6.18% of the tiny amount of CPU used along the yell bar, <0.5% of the overall CPU available on the PC.

The spikes in memory are from the animation processes of all the instruments updating themselves (interpolating the gaps), images always use more memory than simple code (if it’s well written).

Dragonlaird

[Vibstronium]

I would honestly expect the event queue in the operating system to be perfectly adequate for sub-60 Hz events like simulation frames, despite a single offhand mention in an old version of the simulator SDK docs that doesn’t say what they consider frequent.

If you find it makes a difference in practice for recording position information updated every frame, I would be very interested to see your performance measurements to better understand what’s going on.

[Dragonlaird]

I’d also like to understand why it is necessary to record every frame, even then, I see no reason you couldn’t record a rapid stream of values from MSFS/SimConnect using an event-driven approach, with values being cached and pushed to your client (Flight Recorder).

[Steeler2340]

@Dragonlaird

I’d also like to understand why it is necessary to record every frame, even then, I see no reason you couldn’t record a rapid stream of values from MSFS/SimConnect using an event-driven approach,

Well, the only reason so far why I am requesting the aircraft’s position “per simulated frame” is because the PERIOD enum does not offer a wide variety of choices in the first place:

  SIMCONNECT_ENUM SIMCONNECT_PERIOD {
    SIMCONNECT_PERIOD_NEVER,
    SIMCONNECT_PERIOD_ONCE,
    SIMCONNECT_PERIOD_VISUAL_FRAME,
    SIMCONNECT_PERIOD_SIM_FRAME,
    SIMCONNECT_PERIOD_SECOND,
  };

Another alternative would be “per visual frame” (which at least on my computer would probably be lower than “per simulated frame” ;)).

Once per second might be sufficient, too, but I’ll see about that once I have a proper “interpolating code” working (which would interpolate between those 1 second intervals - again, setting the aircraft’s position once per second might already give a "smooth playback experience, I have yet to experiment with all those sample frequencies).

The other reasons why I am sticking to a “polling” approach - for now - are A) I don’t have a hWnd handle, as my code is supposed to work “GUI-less” (but yes, I already have a GUI as well, so I could “inject” the required hWnd handle, in order to access the win32 event queue, if I wanted to), …

… but most importantly also B) really for simplicity and performance reasons: I already know that a message will be there (at least I expect it to be there), so there is simply no reason to wait for the win32 event queue to tell me that a message has arrived. Because I know it will be there, based on the period (frequency) that I have specified in my SimConnect_RequestClientData call.

(But yes, in case of “once per simulated frame” the frequency is still totally unknown, as it most likely depends on the performance of the system anyway. And again, sampling the aircraft’s position may be too high anyway).

So, and just about now I have an A320 to land - you’ll excuse me

UPDATE: Oh, and I fully agree that the “win32 event queue overhead” may totally be irrelevant these days - especially given the fact that FS 2020 already requires a powerful horse in the first place And yes, recording of the position’s aircraft would work both ways: “polling” and “event driven”, I fully agree.

[Dragonlaird]

Here’s a link I found useful when developing a Message Handler without a Windows Form.

c# - Message pump in .NET Windows service - Stack Overflow

For info, I do not set an update frequency in my server component, it simply calls my WndProc method each time a value changes. Which I suspect is all you need for your Flight Recorder.

Although there’s no time-stamp, in the received message, there doesn’t need to be, as it’s the time WndProc was called.

If your code is efficient and Out-of-Process, you can capture the values in real-time, cache them with a time stamp then push/stream them or save them as required, in a new thread, so as not to interrupt the next message.

Just a though to help you

Dragonlaird

[Steeler2340]

@Dragonlaird

Although there’s no time-stamp, in the received message, there doesn’t need to be, as it’s the time WndProc was called.

That’s exactly what I intend to do: I simply create a timestamp just before I call all the “dispatching”. Each received position is then “enriched” with that timestamp (“elapsed time since recording start”).

If your code is efficient and Out-of-Process, you can capture the values in real-time, cache them with a time stamp then push/stream them or save them as required, in a new thread, so as not to interrupt the next message.

I don’t think that in my case a separate thread will be required: I will simply store the received values in a ring buffer (and inform any interested consumer(s) about it). That doesn’t take long, as no further “data processing” is done at that point. And assuming that I will at maximum be sampling at 60 Hz that gives me more than enough time to store the data (specifically: up to 1 / 60 seconds).

But yes, another “worker thread” will then fetch the data from the ring buffer and periodically persist all the data received up to that point (I foresee a simple SQLite database for this - but I shall see whether I will initially go for a simple stream-based file approach).

For now I store everything in memory, and will add the (still missing) timestamps and interpolation code to my little project…

Btw. my landing in the A320 in Gibraltar just came to a stop like 10 metres before the sea

[devsnek]

while benchmarking my rust library I found that CHANGED + SIM_FRAME is ideal in almost all cases. the edge case is very very large structs, which could probably be better served by adding the TAGGED flag, but I haven’t tested that yet.

[poopyBeard86]

Thanks again for your suggestion Vib!

You should be able to create a single hidden window (bare win32 calls should be fine if you don’t want to play with modern frameworks) and pass its window handle and a preferred window message ID from the user area (the examples all use 0x0402) to SimConnect_Open. Then in your window procedure, when you receive that window message call SimConnect_CallDispatch until it returns 0.

Until yesterday i never touched the win32 api and yet this works like a charme. I just opened a new project in VS using the C++ Desktop App template, populated the //TO DO:.. sections with my earlier code according to your description. It was way easier than i thought and this is so much neater than just a sleep in the while loop.
Awesome christmas present, thanks!!!

EDIT: …but know i need to get serial communication running in this event driven environment. Doesn’t seem so easy anymore.

[Dragonlaird]

Don’t worry about stopping on the runway, if it’s any consolation, piloting a submarine is very much like flying a plane, I speak from experience

[Vibstronium]

Woohoo! Good luck with the serial communications, that’ll be the “fun” part…

[poopyBeard86]

Yeah “fun” part… i would rather say i entered a world of pain

i quickly realised, that the serial library i used before, isn’t really up for the job.
so i was looking around if i can find a nice C++ library that is aimed at working together with this win32 message pump and is preferably multithreaded (without me needing to take care of any threads).

first i found this guy:

Serial library for C++

it’s about 20 years old. I got it to compile but didnt get any data transmitted in either direction. and its 32bit.

So i looked further and found this:
https://www.tetraedre.com/advanced/serial2.php
also about 20years old but i got it working right away. It compiled, transmitted and received data, is multithreaded and ready to be used with serial events/message pump. WOOHOO? no.
It’s of course in 32bit. To my surprise i got it to compile in 64bit but it would throw access violation errors when running.

Now i’m not shure where to go from here. I probably should write my own library but writing a nice serial library isn’t that trivial(for me at least).

Porting 20 years old libraries to 64bit is definitely over my head.

Going back to my singlethreaded non-win32 code would be a bit cavemanish and i like a good challenge and to learn something (but everything worked there, including serial communication)

The last option i can think of, is using one of the single threaded 64bit serial libs that are around and implement the win32 serial event functionality myself. That would probably work… maybe
But then again the 64bit serial libs that i could find aren’t as sophisticated as the 20years old ones. bummer.

I feel im running in circles here…

If anybody who reads this has some valuable advice please share it with me desperation starts to kick in…

oh… and sorry if i went too much offtopic

[Dragonlaird]

@AndreZwicky

but know i need to get serial communication running in this event driven environment. Doesn’t seem so easy anymore.

You could revive an old-school approach to capturing serial data and buffering it for use by another component/application.

Many years ago we used FOSSIL Drivers for this, their primary purpose was to standardise the serial interface but they quickly became the standard for caching serial data.

[Steeler2340]

@Steeler2340

@Vibstronium

Then in your window procedure, when you receive that window message call SimConnect_CallDispatch until it returns 0

Are you sure about that? Or asked differently: don’t you get a user “window event” per message? Because according to the (Prepar3D) API docs:

I have made some further findings, at least as far as the “polling” use case is concerned: SimConnect.dll does indeed cache the received messages! I stumbled over this by chance, as when I press the “stop recording” button in my GUI prototype I kept receiving messages - and once I pressed the button again (“continue”) my callback was called for each message that had arrived since I last called SimConnect_CallDispatch()!

Well, maybe not for everyone, as I did not do any further experiments of big the message cache actually is (and it is of no use to me anyway, as I am only interested in the “latest” message, at the point of “data sampling”, and the mesages do not seem to have any timestamp by their own) - but the cache definitively can hold a couple of hundred messages, as it seems (depending on their size possibly).

(I am now setting the period to SIMCONNECT_PERIOD_NEVER whenever the “Pause” button is pressed, in order not to receive any messages in the meantime).

So my logic is now going to be:

• Request the data with a period of SIMCONNECT_PERIOD_SIM_FRAME
• Sample the data, possibly at a slower rate (20 Hz, 10 Hz, 5 Hz, … )
• For every “sample” (triggered by a timer)
  • evaluate the elapsed time (since “start of recording”) → “current timestamp”
  • call SimConnect_CallDispatch
• In my callback
  • keep processing the messages, but
  • discard (overwrite) any previous replies for as long as the “current timestamp” is the same (as I am only interested in the “latest aircraft position” - based on the assumption of course that my callback is called in the same order as the messages have originally arrived, with the oldest message first).

Or in other words: there is no need to iterate (in a loop) over SimConnect_CallDispatch: once per Win32 event (“event based message processing”) respectively once “per polling” (e.g. timer-based) should be enough. The callback is then called by SimConnect.dll for each message that has arrived since the last “dispatching” (possibly discarding the oldest messages at some point in case SimConnect_CallDispatch is not called often enough).

Is my understanding here correct? Did others also make the same experience respectively have a pointer to some SimConnect API documentation where this might be confirmed?

[LoungeFlyZ]

What is strange about the SimVariable example project from the SDK is that it has ALL of the Windows Events stuff setup to handle the sim connect event (WM_USER_SIMCONNECT), but then doesn’t use it for requesting data and does it using polling!

I tried this out by removing the polling and using RequestDataOnSimObject instead and it worked well. In fact it worked much better! Updates were much much faster than polling.

[poopyBeard86]

Hi Dragonlaird
Thanks for the advice. I got it working properly by now with this library:
CSerialport from Naughter

It’s the only serial c++ library for win32 that is working in overlapped mode, compiles straight away in 64bit and is still maintained that i could find. really glad i found it.

[Steeler2340]

For the record: Sky Dolly - a flight recorder - making use of the above discussed topics has just been released in a very early - but functional, as far as record/playback is concerned - preview release, also refer to here:

[ANNOUNCEMENT] Sky Dolly - Flight Recorder | till213

It is open source, so feel free to see how I read/set the various aircraft position and attitude variables (including cubic interpolation for smooth replay, and properly implemented “modulo 180|360 math” ;)).

For now I still use a “polling” approach, however the actually received data is only sampled at a user defined sample rate, or in other words: previously received data points are simply updated (“upsert”) for the given timestamp.

I might try the “event approach” in a later phase of the project. But the polling approach works quite well for now, and CPU usage during recording is < 1% (on my i7 4.2 GHz system anyway ;))

[Dragonlaird]

@Steeler2340

Or in other words: there is no need to iterate (in a loop) over SimConnect_CallDispatch: once per Win32 event (“event based message processing”) respectively once “per polling” (e.g. timer-based) should be enough. The callback is then called by SimConnect.dll for each message that has arrived since the last “dispatching” (possibly discarding the oldest messages at some point in case SimConnect_CallDispatch is not called often enough).

I found the native Windows Message event listener is called whenever anything wants to raise an event, not just SimConnect.

I know the .NET code hides most of this handler code in lower level classes (not really relevant for you I know) and consumes them invisibly, however it’s relatively simple to write your own handler to filter out unwanted messages and then raise an event in your own code whenever SimConnect has something to say.

Below is a snippet of code I use for capturing these Windows Messages and raising an event whenever something of interest appears. It should be fairly easy for you to convert to C++. It is lightweight and not resource intensive.

I run my code within a task, so it is permanently running (but can be disposed of, as tasks are easier to cancel than a thread).

internal class MessageHandler : NativeWindow
{
    public event EventHandler<Message> MessageReceived;
    private const int WM_USER_SIMCONNECT = 0x0402;

    public MessageHandler()
    {
    }

    internal void CreateHandle()
    {
        CreateHandle(new CreateParams());
    }

    protected override void WndProc(ref Message msg)
    {
        // filter messages here for SimConnect
        if (msg.Msg == WM_USER_SIMCONNECT && MessageReceived != null)
            try
            {
                MessageReceived.DynamicInvoke(this, msg);
            }
            catch { } // If calling assembly generates an exception, we shouldn't allow it to break this process
        else
            base.WndProc(ref msg);
    }

    internal void Stop()
    {
        base.ReleaseHandle();
        base.DestroyHandle();
    }
}

Then I hook into the event handler and my listener is called every time SimConnect notifies me of a change, then within my listener I call the SimConnect process to fetch the message.

/// <summary>
/// Every Windowws Message is captured here, we check for SimConnect messages and process them, else we ignore it
/// </summary>
/// <param name="sender">Windows Object generating the message</param>
/// <param name="msg">Message from sender</param>
private static void MessageReceived(object sender, Message msg)
{
    if (msg.Msg == WM_USER_SIMCONNECT && simConnect != null)
        try
        {
            // SimConnect has something to tell us - ask it to raise the relevant event
            if (simConnect != null)
                simConnect.ReceiveMessage();
        }
        catch// (Exception ex)
        {
            // Seems to happen if FS is shutting down or when we disconnect
        }
}

[Dragonlaird]

@LoungeFlyZ

I tried this out by removing the polling and using RequestDataOnSimObject instead and it worked well. In fact it worked much better! Updates were much much faster than polling.

Yes, the example project is only intended to provide you with enough to get started.

You are correct when you mention it uses polling and that it can be done much faster, there are 2 methods within SimConnect to request variable updates, the first is to supply an update whenever it is requested, the second supplies updates based on the frequency selected.

The latter option I find is too restrictive in the available frequency options, it can provide an update every:

• Never
• Once
• Second
• Sim_Frame
• Visual_Frame

But if you want to request a value update, say every 50ms, you can’t, you need to use the polling option via a timer event.

[Dragonlaird]

@Steeler2340

I don’t think that in my case a separate thread will be required: I will simply store the received values in a ring buffer (and inform any interested consumer(s) about it).

Yep, I see no reason you couldn’t update your own ring buffer in this way, however, the “inform any interested consumers” did remind me of a couple of potential pitfalls.

Soo, with that in mind, here are a couple of quick tips, in case you fall foul problems that I’ve encountered in the past.

If you’re triggering an event handler to notify interested parties of the update:

1. Ensure you capture any exceptions when invoking the event handler, as the listener(s) are called in-process. If the consuming code has an error, it can cause the event handler to raise an exception in your own code.
2. Invoke the event handler within a task, rather than in-process. If the consuming code is poorly designed, it can take a while for your event handler to complete, potentially slowing your own code or causing a race condition that ultimately causes an exception, if it’s in a self-contained task, you don’t care if the consuming code is slow or causes an exception.

Just my two pennies worth in case you hadn’t already covered the above in your own code.