Mixed reality

[BaronVonVaderham]

Hey all,

 I’m contemplating getting a vr kit later in the year, but it’ I’m also contemplating making some cockpit panels and consoles. I’d imagine that vr combined with real panels didn’t make sense, but at work we use mixed reality where we can see the panels but also have the vr benefits. 
had anyone experimented with that?

[dburne]

If you are really interested in VR, suggest you start out with a VR headset first and see how you like it.You may find your desire for panels to diminish.

[MADLOU1]

2 hours ago, BaronVonVaderham said:

Hey all,

 I’m contemplating getting a vr kit later in the year, but it’ I’m also contemplating making some cockpit panels and consoles. I’d imagine that vr combined with real panels didn’t make sense, but at work we use mixed reality where we can see the panels but also have the vr benefits. 
had anyone experimented with that?

I would love to see how that works, being able to see real panels outside of the VR headset while being able to still see the VR world.  Best of both worlds!

[BaronVonVaderham]

Hey dburn,

 you’re right, and it’s likely going to be my direction. I had a taster with VR on PS4 ace combat, and it’s an incredible immersive experience. It gives a definite advantage in aerial combat. 
maybe in due course when displays become more affordable I could build a cockpit surrounded with displays, which should give a comparable experience. 

[Nodak]

Or you can wait and see what Extended Reality -XR brings to the table, which allows for high res pass through and blending. 

Varjo is leading the drive so far, but sony and samsung have XR sets soon to be revealed.

https://www.youtube.com/watch?v=GS1426PlQuQ&t=130s

Too see it practically applied.

[Toge]

Check out VR Reality Mixer in Steam: https://store.steampowered.com/app/1844610/Reality_Mixer__Mixed_Reality_for_Vive_and_Index/ and

 

Costs 4 euros and allows you to use the cameras of some common VR kits for mixed reality. Just set my Reverb G2 up with it (beta version supports WMR headsets). It works beautifully and allows me to see and use my physical MFD buttons etc. The picture quality of e.g. the G2 cameras is blurry black&white, so you won't be able to read small text etc, but will find your buttons and switches.

For now it only supports SteamVR though.

[zetikka]

On 4/22/2023 at 3:26 PM, BaronVonVaderham said:

Hey all,

 I’m contemplating getting a vr kit later in the year, but it’ I’m also contemplating making some cockpit panels and consoles. I’d imagine that vr combined with real panels didn’t make sense, but at work we use mixed reality where we can see the panels but also have the vr benefits. 
had anyone experimented with that?

Playing these days with MR on DCS, using Meta Quest 3 + Virtual Desktop + Openkneeboard on the DCS OpenBeta (OpenXR). Seems promising (very early stages of testing):
 

Note that Openkneeboard was test to auto switch on/off depending on the area I looked at, it explains why the MR grid goes on and off.
 

Main limitation at this point is that you can only open a single-plane chroma compositing window for passthrough video, while (in this case) Hornet's cockpit displays are at multiple focal planes, which results is paralaxing on the VR zones I left for MDF/UFC retention.  

Edited February 14, 2024 by zetikka

[zetikka]

Once Helios gets enabled while in VR (a bit tricky) it starts getting interesting. DCS treats the main and secondary monitor as a single display and inserts the viewports in it, so we get part of the animated cockpit as helios background. Static elements can be hidden by panels, but RWR and IFEI retain a transparent background and look a bit messy. 

 

[Thundersmoke]

On 2/14/2024 at 12:50 PM, zetikka said:

Once Helios gets enabled while in VR (a bit tricky) it starts getting interesting. DCS treats the main and secondary monitor as a single display and inserts the viewports in it, so we get part of the animated cockpit as helios background. Static elements can be hidden by panels, but RWR and IFEI retain a transparent background and look a bit messy. 

So I have the WinWing MFDs & displays (x3) and I've been trying to export the F-15E screens.  Works fine in 2D just using the winwing software, but in VR, the MFDs have the transparent backgrounds and are more or less unusable.

Is this why, and are Helios graphics the way to block out the video?

Edited February 17, 2024 by Thundersmoke

[actually_fred]

On 2/14/2024 at 6:02 AM, zetikka said:

Playing these days with MR on DCS, using Meta Quest 3 + Virtual Desktop + Openkneeboard on the DCS OpenBeta (OpenXR). Seems promising (very early stages of testing):
 

Note that Openkneeboard was test to auto switch on/off depending on the area I looked at, it explains why the MR grid goes on and off.
 

Main limitation at this point is that you can only open a single-plane chroma compositing window for passthrough video, while (in this case) Hornet's cockpit displays are at multiple focal planes, which results is paralaxing on the VR zones I left for MDF/UFC retention.  

 

This is a hardware limitation; with the possible exception of the Varjo XR4 (the advertising material is... advertising material, not clear functional descriptions), no generally available headset is capable of rendering anything at multiple focal planes.

[zetikka]

On 2/24/2024 at 4:24 PM, actually_fred said:

This is a hardware limitation; with the possible exception of the Varjo XR4 (the advertising material is... advertising material, not clear functional descriptions), no generally available headset is capable of rendering anything at multiple focal planes.

I think my statement lacked accuracy and was misleading: When mentioning the need for several focal planes, I was thinking about player who have physical center panels with MDF placeholders (like Winwing Phoenix), not my personal setup which is based on a single touchscreen and Helios. 

In my case there is no possible way I could match in the depth space my single-plane touchscreen with virtual MFDs, which are at different distances and angles.

The case of the physical panel is different, since you want all of the panel visible in passthrough video, excepted for the surfaces of the three MFD plus the UFC display and radio 7-digits, which happen to be at different depths of each other from the VR PoV. 

If you have been using Openkneeboard in VR you know that you can set the board at any depth and angle from your PoV you want. Since that board is our chroma compositing plane for video passthrough, if you "stick" it to the plane of your MFD and cut out a piece (alpha channel =0) just the apparent size of the MFD then you really have a passthrough view of you physical panel with an aligned "from VR" MFD display.

You can already do that for just one MFD (well... actually two, if you set it for right and left MFD) since Openkneeboard allows you to use a single board (compositing window, for us). If it would allow to use 3-4 boards simultaneously, each placed at an individual depth/angle in 3D space corresponding to a "to retain VR element" plane, then - ideally - you would end up seeing the whole of your physical panel with just the from-VR display inserter at the right places.

Now this is *theory*, because it would require that the physical panel's distance to your headset cameras, depth deltas between elements (think UFC and MFDs) and angular size do match what ED has modeled in the Hornet's VR cockpit. I am not betting a lot of money on that when I compare my physical cockpit stick/hotas/dudder position to the in-game VR ones...

Bottom line: I guess we will probably get better resolution passthrough VR headsets before ED implements the option to position in the VR space actual viewports 

[zetikka]

On 2/17/2024 at 3:07 AM, Thundersmoke said:

So I have the WinWing MFDs & displays (x3) and I've been trying to export the F-15E screens.  Works fine in 2D just using the winwing software, but in VR, the MFDs have the transparent backgrounds and are more or less unusable.

Is this why, and are Helios graphics the way to block out the video?

 

As I understand it the issue in more on the ED side than on Helios'. The only way to get Viewports exported in VR is to check that "use system resolution" box in the VR tab (quoting from the top of my head), which will re-copy the VR view to all your physical monitors, including the one you use for Helios. I have added masking areas (actually black polygons) to my Helios template to hide most of the background image, but for some reason ED does not include an opaque background to the UFC, IFEI (radio) and RWR ViewPorts - while it does for the MFDs:

So I am as stuck on it until ED changes that...

[Thundersmoke]

11 hours ago, zetikka said:

As I understand it the issue in more on the ED side than on Helios'. The only way to get Viewports exported in VR is to check that "use system resolution" box in the VR tab (quoting from the top of my head), which will re-copy the VR view to all your physical monitors, including the one you use for Helios. I have added masking areas (actually black polygons) to my Helios template to hide most of the background image, but for some reason ED does not include an opaque background to the UFC, IFEI (radio) and RWR ViewPorts - while it does for the MFDs:

So I am as stuck on it until ED changes that...

Ah, good to know.  I think I've given up on the mixed reality for the moment until Somnium or Varjo knock my socks off with a next-gen offering.  I've since discovered Voice Attack and profiles that allow me to operate the MFDs with my voice.

If my wife walks past at the wrong time, she thinks I'm playing a very expensive game of Battleship.

T5, right

B2, right

R6, right

L3, left

etc.   Lol

But it's a little painful to have the MIP setup and now not really be using it.  So I'm looking forward to the VR1 or XR4.  My credit card isn't.  But I am.

[zetikka]

5 hours ago, Thundersmoke said:

Ah, good to know.  I think I've given up on the mixed reality for the moment until Somnium or Varjo knock my socks off with a next-gen offering.  I've since discovered Voice Attack and profiles that allow me to operate the MFDs with my voice.

If my wife walks past at the wrong time, she thinks I'm playing a very expensive game of Battleship.

T5, right

B2, right

R6, right

L3, left

etc.   Lol

But it's a little painful to have the MIP setup and now not really be using it.  So I'm looking forward to the VR1 or XR4.  My credit card isn't.  But I am.

I share your pain... also using VoiceAttack as main VR interface, but gets a bit messy when you fly multiplayer and add SRS, and possibly Discord voice as well to the mix.  
 

[zetikka]

I just dropped a short video on last advances in my Mixed Reality project with the Quest 3 - this thing is really taking shape. Since last video I added some Winwing physical panels and refined the compositing thanks to the new options in Openkneeboard.  Hope it will inspire you for your own builds!
 

 

[Dogmanbird]

Nice Zetikka!  It's a slow trip, but we're getting there