So I’ve toyed with different replay tools over the past week and there are some things that reared their ugly head.
1)Stutters. You’re either going to get stutters from recordings at graphically-intensive areas if you use one tool, or you get fluttering wings and smoother performance with another. I prefer stutter that I can edit out as opposed to flutter which really impacts the visuals throughout the recordings every time a frame skips or some kind of momentary lag happens.
Your choice- pay for one problem or get the other for free. It is what it is.
Cockpit states don’t replay- so if you’re flying a jet or anything with a route/AP- it won’t show those states in the FMC or cockpit. Altitude, alitmeter, airspeed- all have to be reset if you want to capture them accurately in the replay video. You’ll also get bells and alarms in the cockpit and on the recordings unless you’re taking external shots. Same happens if you rewind the replay without starting a fresh flight and positioning the replay slider exactly where you want it. Same with altitude call outs.
Lights will go on and off as switches tick-tick-tick over. and. over. again. If you happen to turn on a landing light that gets captured by the recording, chances are that it will loop during replay- the lights also flicker on and off repeatedly.
Same with engine sounds. Many times in the 320, 787 and 747, the engines will rev up to full power and stay there. Pull back on the thrust and they go right back. Kinda stupid to hear on final approach.
Pushback utilities don’t show up- no tugs, no nada. Sometimes the engine startup is also messed up. I’ve read numerous times that we shouldn’t start the replay until AFTER the engines have started and everything in the cockpit is configured.
Flight controls can work backwards sometimes- like ailerons moving in the opposite direction they’re supposed to be moving.
Doesn’t record weather- soo…if conditions changed between the time you recorded the flight and when you played it back- factor that into your expectations.
Some of these things are a big deal, some are not. I imagine once the gazillion limitations in the sim in its current state is are overcome, replays will get more-sophisticated over time. My guess is this is one reason that Asobo isn’t too enthusiastic about rolling one out just yet. There are too-many data points that will not get recorded or they’re just fouled up. Not-too-mention, that kind of stuff doesn’t paint the sim in a good light on YouTube either…
Of course, there are plenty of ways to work around this stuff, with some good editing and movie-magic, but things like recreating full flights and such will take a lot of time and effort to produce.
On the other hand, these replay tools are awesome for scenic shots and other bits that don’t require a lot of info from the sim…external takeoff/landing shots, acrobatics, etc- but for heavy jet flying, all the tools out there fall short of what we really need to make it work.
I’m enjoying replays- but it’s important to remember all tools are very limited at the moment, and we’re a long way from being able to simply fly from point a to b in a tubeliner or with APs engaged and record every aspect of the flight.
“Ugly head” is a bit of a strong word here I’d say that most tools are still in their infancy - and so is the provided “SimConnect API”, specifically its implementation state in FS 2020. E.g. you don’t have access to multiplayer aircraft (only “AI controlled aircraft and boats”), the whole “weather API” is non-existing and even declared “obsolete” (due to the radical new weather system in FS 2020, as opposed to its original implementation in FS X etc.).
As a developer of such a recent “recording tool” I can share some technical insight and my personal thoughts - if that is of interest to you here.
So I’ll go through and try to answer some of your raised points below.
The following points have a direct influence on a smooth playback experience (or the lack thereof)
Interpolation of the recorded (“sampled”) data points (“simulation variables”)
Amount (and frequency) of data sent from the SimConnect client (“recording tool”) to the server (“FS 2020”), and specifically how fast the server is able to process (“playback”) it
The simulator itself which still tries to apply its own “physics”
Point 1) is easy to solve. Some here use a “proprietary algorithm”, others - like me - simply call it “basic math”. Specifically when it comes to a smooth interpolation of a given set of sample points.
From the Nyquist-Shannon sampling theorem we know that we can perfeclty reconstruct any curve with a given frequency when we sample at “double its frequency”.
Ever wondered why music CD have a maximum sampling frequency of 44.1 KHz? Exaclty: because we can perfectly reproduce any audio signal with a maximum frequency of 22.05 KHz - the “half of 44.1 KHz”. That’s exactly an application of the Nyquist-Shannon sampling theorem.
Now you may wonder why music production software lets you sample up to 96 KHz, and even with 24 or even 32 bit per sample, as opposed to the final 44.1 KHz with 16 bit per sample on the final music CD: that’s because during production you mix a lot of signals, which may both produce aliasing effects when you overlay (“mix”) signals with almost the same, but slightly different frequencies - simplified - and may result in “clipping” if the added loudness exceeds the range of 16 bit. Only in the final mix (“mastering”) you then “downsample” to 44.1 KHz with 16 bit, taking the maximum “loudness” (maximum sample value basically) into account.
But I digress
Now Flight Simulator does not give us twice as many sample points as its calculated - and displayed - frames per seconds; we wish
So at best we get the same amount of sample points as the current frames per second count, but as we know the FPS may vary over time. And FS 2020 may otherwise be a bit slower with sending the sample points to the client, as this may have lower priority than keeping up a constant FPS.
In fact, I believe that the original SimConnect API was mostly invented for “external hardware controlllers and displays”, and they are fine to update their panels with a way lower frequency than the current FPS - but not necessarily for “real time sampling of all simulation variables”.
So what do we do if we have - or even want to have, in order to minimise storage requirements - less sample points (say 10 Hz, or 10 sample points per second) than the FPS we aim for (say 60 Hz, or 60 frames per second)? We interpolate.
As mentioned my recording tool uses basic math, cubic spline interpolation (Hermite interpoation) to be specific. So at least the requested sample points (“curve”) is smooth along its path.
(My application also properly takes the proper “modulo 180/360 math” into account - other tools “glitch” there, as they consider e.g. the difference between a heading of, say, 359 and 0 degrees to be 359, when in fact it is just 1 degree of difference - except in the unlikely case that your aircraft really does an almost 360 degree turn within one frame, of course ;)).
For point 2), we can try to send the interpolated data as fast as we want (my tool let’s you choose that playback frequency as well, separate from the recording frequency, or “sample rate”), but there is probably a limit of how fast FS 2020 is able to properly reproduce those requested sample points and render them on time. That limit is basically limited by the capabilities of your hardware, but also the efficiency of the implementation in FS 2020 itself, read: “outside of the reach of the actual record/playback tool”.
For point 3), any decent record/playback tool will “freeze the physics” during playback that influence latitude/longitute, altitude and the attitude (bank, pitch, roll) of the aircraft. But there might still be other forces (wind perhaps) that FS 2020 might still try to apply to the aircraft “overall position/behaviour”.
That all said, with my tool I get a decent playback experience, especially noticeable during “super slow playback” (at say, 1/10 of the original speed). But I do get the occasional “stutters” as well, but I currently attribute them to my limited hardware (yes, an iMac 27" from 2017 with 32 GB RAM and an older Radeon Pro 580 ;)), as also during “normal flight” I do get the same stutters every then and when (especially if geometry is dynamically loaded).
It is both a question what a replay tool tries to achieve and also trying to limit the amount of exchanged data (which not only influences memory, but also processing time on both the client (“recording tool”) and server (“FS 2020”).
In my case I just started with the most basic “simulation variables”, especially those which have “visually the greatest impact”. What do I mean? I refer to aircraft controls such as rudder, flaps, ailerons, spoilers etc., all which have a “visible effect outside of the aircraft”. Because my assumption is that people who want to record their flights for “YouTube video production” mostly show external views of their aircraft.
Or in other words: my current goal is not a 1:1 reproduction of the entire flight.
Also, not every simulation variable is readable, let alone actually writeable. Especially when it comes to all the “glass displays” of all the various aircraft (the MCU of the A320neo comes to mind). And trying to store all possible simulation variables - especially “in real time” - would probably exceed certain performance limits as well.
There are two things to node here: for one, certain “simulation variables” are readable, but not settable (for unknown reasons). This is specifically true for all (external) aircraft lights, see e.g. here (official SimConnect API documentation for FS 2020):
So what do you do? You trigger lights as events (as opposed to simulation variables).
Now here’s the deal: whereas you record simulation variables for each sample point you record events only when they occur. Or in other words: when you randomly position the playback position on the timeline you do have the current (interpolated) state of all the simulation variables at that point (e.g. flaps position, rudder position, and of course the position and attitude of the aircraft), but not the events - unless you’d “scan forward or backward” and check “what the next/previous event” was.
And especially if you “scrub backwards” in the timeline you would have to “undo” certain events. That might be easy for “boolean events” (on/off), but some events are involve “non-trivial values” where the “reciprocal value” might not be evident. Or in other words: it might not be worth the effort to reliably restore the “event state” for any given (random) point on the timeline. And that’s probably why most record tools don’t bother.
For your second point, the “repeated clicking” respectively setting “the same value over and over”: while my own recording tool does not yet implement events (specifically for lights) I noticed the exact same problem with the “flap position”. That value is a simple, discrete value between 1 and N, where N is the number of "flap positions). And even while I am sending the exact same value, say 3, for a series of sample points FS 2020 erratically still switches the flap position between 3 and some other value, and back again to 3. That happens very quickly (and provably I am always sending the simulation variable as 3), so this might simply be some bug in FS 2020 itself.
Of course I cannot yet speak about the lights (which are “triggered via events”), but I could imagine that it might be a similar problem there as well.
Or in other words: nothing much a recording tool can do here (except maybe trying to find some workaround?).
In my recently released version 0.2 of my recording tool I just implemented “basic aircraft controls”, including the engine throttle positions (for engines 1 to 4). So what a recording tool basically does - again - is record and playback the throttle position and send it to FS 2020 during playback. It is the “physics engine” in FS 2020 itself which then reproduces the sounds accordingly.
I haven’t paid much attention to “how much delay of sound” there is when changing the corresponding levers, because as I said, I just implemented this yesterday
While I haven’t even touched those areas this is probably all due to a limited functionality of the SimConnect API itself. E.g. I don’t remember having seen any functionality related to the pushback tugs. You can get “fascilities data” = basic runway / airport information such as “where is the fuel station”. But again, not sure about pushback tugs, let alone control (“set”) them.
Also, I read from some other tool that it is surprisingly non-trivial (or even impossible at the current stage) to detect whether the engine(s) has started or shutdown.
Most flight controls let you either record their actual position (e.g. flaps in degrees), or their corresponding control setting (flap position, 1, 2, 3 etc.). Currently I record the actual “control handle position” rather than the actual flap position, whenever possible. It is then the simulation itself which sets the flaps (and ailerons, elevators, as I merely record the "yoke position which controls them).
Maybe other recording tools record the actual control surface positions, but even then I couldn’t imagine why they’d move “in the opposite direction” (if properly implemented/sent as request data).
The entire SimConnect API related to “weather” is not implemented at all in FS 2020. In fact, it is marked “obsolete” (= the existing API won’t be implemented in its current form), as the weather simulation has fundamentally changed (since the days when the SimConnect API was first introduced).
So this is simply something we have to wait and see how this develops.
As mentioned at the beginning the focus of my tool will be “visually appealing flight recordings”. For as long as the - external - control elements move and act “according to the flight movements” (respectively vice versa, of course ;)) I have reached my own design goal.
That said, I also slowly add more and more variables as I see fit, having a close eye on potential performance (decreases).
In software engineering adding is always easier than removing.
Personally I am mostly looking forward to an API which let’s us actually control the camera As far as I understand this is still not available (implemented) in FS 2020.
Now I realise that my post has become a bit of a lenghty excourse into add-on/tools development. But I hope that some of my thoughts were of interest to you and anyone else, and hopefully they shed some light on where (and why) we currently are with regards to tool development.
This is, by far, the most-enlightening post I’ve read in here in a long time. Thanks for that. Which replay tool is yours? I’ve tried a couple now, and both pretty much do the same thing, but I’m happy to try anything else to see which one works best on my system.
The one which just got released in version 0.2 yesterday evening
I must admit that I just discovered an embarassing bug while producing the following video, which I just made public a couple of minutes ago as well:
Specifically the new feature - recording of the ailerons, elevators, rudder etc. - does not work, I am wrongly rounding the recorded values to integer values (due to a last minute adjustment of the data type, from the previous “int16” to double. Oh well. So much for releasing late Saturday evening
It is already fixed, so the upcoming v0.3 will properly replay the ailerons etc.
That’s awesome. I had no idea you were working on a utility too. I’ll check it out. The ailerons have been a big deal for me because I like to take shots of pax views, especially for takeoff and landings, so turns are kind of irritating on that front.
If you are talking about Flight Recorder, this is fixed . A bit hard to read when you don’t want to name names.
Do you encounter this on anything other than the 32NX? I really want to solve this but this seems to happen randomly and only on the 32NX (for me).
This is why people shouldn’t rush Asobo on this. They said there replay needs a separate computer, so they seem to record everything, which is no easy task. The wild west of 3rd party aircraft all wants to do different things doesn’t help the situation either. If we want replay feature now, we definitely get a much simplified feature.
I’m not avoiding naming names- all replay tools I used had the same problem, and I’ve used yours a lot. The 787 and FBW 320 (all versions) have the reversed flight control issues and cockpit sound glitch. However, I know there are a couple of devs who are working on fixes as we speak.
I planned on uploading more videos shot with replays to compare their performance, but these small issues have led to a lot of editing, and I don’t have a lot of free time for that. I don’t want to put up videos that show all the glitches, otherwise people will only complain about the problems and not the good things. When they get improved, I’ll do more:-)
I can understand why we shouldn’t rush Asobo after seeing the limitations. However, I’m extremely delighted that you all are developing and improving tools. What you guys come up with will be much better than what Asobo will give us.
For the record: Sky Dolly v0.2.1 (updated yesterday, now also available at https://flightsim.to/file/9067/sky-dolly) fixes the issue with the aircraft controls (ailerons, rudder, elevator etc.).
I tried the latest version. When the plane is on the ground, the ailerons work okay. When the plane is In the air, they are still wrong. Turn left- ailerons on left wing go down and ailerons on right wing go up. Turn right, ailerons on right wing go down and ailerons on left wing go up.
That is not yet implemented: my focus was and is to first get a smooth replay (I am there now) and be “feature complete” with regards to simulation variables (“lights” which are to be received and set as “events” are next on the to do list).
And I want to finalise the data model first before coming up with a persistence layer. Otherwise old files could not be read anymore (unless I would “migrate the data” each time, which is something that I want to postpone / minimise for as long as possible during development).
So as a v0.2 this is still to be taken as a very early preview - but as in “release early, release often”
Sky Dolly is also available here since a couple of days:
(same ZIP binary distribution as on github). There a bold statement in the description explains that persistence is still to be implemented.
Preliminary (as in „format may change / become (partially) unreadable in upcoming preview versions“) CSV import and export is now available in the latest 0.3.0 version.
I’d say that most tools are still in their infancy - and so is the provided “SimConnect API”, specifically its implementation state in FS 2020. E.g. you don’t have access to multiplayer aircraft (only “AI controlled aircraft and boats”), the whole “weather API” is non-existing and even declared “obsolete” (due to the radical new weather system in FS 2020, as opposed to its original implementation in FS X etc.).
. A bit hard to read when you don’t want to name names.
