iPad kneeboard w/ Apple pen in VR?

Is there any way whatsoever to get an iPad and Apple pen to show up in VR? I’d love to have this on my knee, and show the pen cursor while I’m in VR so I can know where the tip is, and then have it stream real time what I’m drawing.

Right now, kneeboard solutions with mouse-based “pens” are too clunky and slow to actually use meaningfully to copy anything from ATC. And with the headset, it’s a PITA to lift it to write stuff down. I want the pen “hover” cursor on the board, and board input displayed live in an iPad window that works in VR.

UPDATE FOR TOP-LEVEL VISIBILITY

1. Create a new Google Jamboard on my iPad w/ the Jamboard app
2. Open my browser to a tablet-sized window on my desktop and load the same Jamboard
3. Pin that Firefox window in Oculus Dash to my “knee” area
4. Launch MSFS
5. Writing on my tablet appears in the browser in VR

A couple problems though:

- Jamboard seemingly only supports landscape view, although you can zoom in to fake a portrait view
- Inconsistent Oculus dash performance with MSFS leads to instances where I cannot access Dash while pushing the button on the Q2 controller. The window periodically disappears as well, so this means if it disappears, it is possible I cannot access Dash again to re-open it without quitting MSFS
- I had an issue with it flickering in and out unless I was alt-tabbed to the browser window (while still in VR)

All in all, it kinda worked, but I’m sure I can do better.
1 Like

This would be so nice to have. Maybe someday…

This morning I thought this post was a bit crazy.
But then by coincidence this evening Fenix has stated that we will be able to draw annotations on their A320 EFB, and also draw using any networked iPads… So potentially you would be able to draw on your physical iPad with your physical pencil and it would appear on the efb in VR . Not exactly what you described but a step towards it!

Well that is sort of nuts. Wish they’d release that as a standalone VR widget. I have zero desire to fly airliners. There’s literally nothing out there I’m aware of doing this.

The video didn’t show it so unclear from the post if you could write with an Apple pen or something on the iPad and have it sync. That would be the functionality I’m looking for.

1 Like

I’m using somewhat similar approach, but with a Wacom 4x6" pen tablet, mapped to a second monitor, which I bring into MSFS with Desktop app in WMR. I open PDF app with pen annotation right on the charts. I can also put all Windows apps there that I need to view, like VATSIM client or Jumpseat radio etc.

1 Like

I see most of the suggestions for WMR or SteamVR, but not for Oculus. I get a major lag fest when trying to pin an app window…

Bump? Anyone successfully get an Oculus app pinned in VR while not making it a lag fest?

I used the VRK tool a while ago but this is SteamVR only. Can you explain how you can bring windows into VR and what tools do you use? I also have some Wacom tablet but it’s collecting dust lately…

So, a hacky way that semi-worked tonight was:

  1. Create a new Google Jamboard on my iPad w/ the Jamboard app
  2. Open my browser to a tablet-sized window on my desktop and load the same Jamboard
  3. Pin that Firefox window in Oculus Dash to my “knee” area
  4. Launch MSFS
  5. Writing on my tablet appears in the browser in VR

A couple problems though:

  • Jamboard seemingly only supports landscape view, although you can zoom in to fake a portrait view
  • Inconsistent Oculus dash performance with MSFS leads to instances where I cannot access Dash while pushing the button on the Q2 controller. The window periodically disappears as well, so this means if it disappears, it is possible I cannot access Dash again to re-open it without quitting MSFS
  • I had an issue with it flickering in and out unless I was alt-tabbed to the browser window (while still in VR)

All in all, it kinda worked, but I’m sure I can do better.

If Virtual Desktop gave better performance and sharpness than Airlink these days, I’d switch back and see if OVR Toolkit could do it, but to my knowledge, it still is behind.

I’m going to try notebookcast.com next and see if that’s a better experience.

Does VRK not work over Airlink then?

1 Like

So the underlying issue to solve with everything when using Oculus Dash is that while the windows appear, and can be pinned (I successfully pinned both vPilot and the Firefox window w/ a Google Jamboard tab open that was syncing w/ the open page on my ipad), the pinned apps keep flashing in and out when MSFS is in focus.

It is VERY distracting. So while I could technically move them really low to my lap and out of sight, then alt-tab when I need to look at them to stop the flashing, it sort of defeats the purpose.

If anyone has any suggestions or ideas as to what might be causing this, fixing this could unlock an easy free VR overlay for all Oculus pilots.

  1. I mapped my Wacom to my second monitor in the Wacom software. Second monitor resolution is reduced to 720p for manageable size inside MSFS. Open any app you want - I use free PDF annotation app from MS Store.
  2. I mapped one of the pen buttons to “Win+Y” keypress so I can transfer window focus back to MSFS (may be unneeded with VR controllers or hand tracking)
  3. I use standard Desktop app in WMR to bring my second monitor into MSFS with controller. Click on the top left icon to select monitor, disable “auto-switch monitors” in the bottom left, then select second monitor. Top right icon for “follow me”, then position to my left for convenience, then click “follow me” again to pin it in place.

I can then write on PDF charts, then click pen button to switch window focus back to MSFS. I wrapped a velcro strip around the pen, and glued one on the tablet’s frame, so I can attach the pen there when not needed and easily find it again when it’s needed. IT’s important, especially considering I have a motion cockpit, so if I place pen somewhere - it won’t stay there unless attached.

1 Like

I have better performance and quality with virtual desktop. You can also now tweak the sharpness and gamma. If you have a 30 series card set encoding to hevc for best results.

So I’ve had it on hevc forever on my 3080. They had a sharpness update, but even with that it still lags behind the Airlink link sharpening in my experience. Also, framerates seemingly lag behind. Am I doing something wrong?

And randomly everything worked fine in Airlink again with both vpilot and a browser window open with Chartfox. Now I need to experiment and go for consistency. This could be the ticket!

I don’t do anything special with virtual desktop:

Codec HEVC
Bitrate 150mb (disable variable bitrate in the streamer)
Graphics quality High or Ultra (This sets the resolution, I used to use Ultra but now tending to use high with extra sharpening)
Gamma 0.8
ASW disabled
Sharpening to your preference, somewhere in the middle
Sliced encoding on, video buffering on
120hz Streaming refresh rate
MSFS 100 render scale
Steam VR 100 render scale
Default Nvidia control panel

Maybe I am doing something wrong with Link though, if you are seeing it considerably better than VD… What settings are you using on link please ?

Link sharpening in ODT on, and then this is my OTT settings.
image

and my Oculus device settings:
image

And Nvidia settings
image
image
image

Would love for you to give it a go and tell me if you find this noticeably sharper than Virtual Desktop.

OVRToolkit and the Steam pinned overlays were non-starters performance wise. Could not even have the game playable.

Today, the Oculus pinned overlay went back to the ■■■■ blinking, but performance was totally fine. I’m not sure if it is me having my SS set too high and a performance limitation or what. If I could crack that nut, it seems like the way to go.

1 Like

I cut the section of face guard around the bridge of my nose out of my goggles. I can see my iPad fairly well.

Ah so you are running quite a lot of supersampling (1.3) and getting some of the performance back with the reduced FOV. I can’t reduce the FOV myself, I find I see the edges of the screen especially on head turn, and this edge is moving at c25fps so looks a bit stuttery.

I had a very quick look and would say I have similar cockpit text sharpness between your settings and VD ultra, but perhaps slightly sharper scenery with your settings (as to be expected with resolution of 3500 vs 2800). The difference does not seem substantial to me though, maybe helped by the VD sharpening. However note you cannot reduce FOV with VD (or increase super sample with the extra performance) so maybe best to stay with your current setup.

My guess is we may be running into the limitations of the Q2 headset when trying to increase cockpit readibility further at these resolutions !

I’m honestly shocked by this. They are night and day to me. Is there a good way to screenshot the differences that would show up?

I think if you hold the right Oculus button there’s a button for a screenshot? Not sure where it’s saved or what quality though.