OpenXR's hand-tracking and controller emulation. Works for anyone ?

[ZerXen]

Hi Gents, I own both G2 and Quest2 and since DCS now has native OpenXR, I really really wanted to play with Quest2's hand tracking ability that OpenXR can nicelly pick-up (The ctrl-f2 menu->inputs->controller emulation). I can nicelly see my hand projected, I can see that the detection system is detecting gestures (in the openxr overlay all those orange debug text). 

BUT

DCS completelly ignores this hand, even if I use the "OpenXR Toolkit Hand-to-Controller Configuration tool" that translates these gestures to an emulated controller button presses. E.g. wrist press should emulate controller gripp button pressed that activates the laser pointer in DCS VR. 

Does this work for anyone? Or is this something ED needs to allow/implement. From my perspective this is like having a leap on a cheap and if working would probably move me back from G2 back to Quest2. 

PS: MSFS actually supports this. 

A quick image how OpenXR projects a fully animated hand to VR

[mbucchia]

This mode is primarily designed MSFS and I don't expect it to work well in DCS.

Instead, look into @actually_fred's HTCC: https://htcc.fredemmott.com/

[ZerXen]

2 minutes ago, mbucchia said:

This mode is primarily designed MSFS and I don't expect it to work well in DCS.

Instead, look into @actually_fred's HTCC: https://htcc.fredemmott.com/

I tried the HTCC, but that is really just a immersion breaker for me going back to a mouse-like pointer. I was really looking for trying "leap motion"-like experience here because essentially all the hand skeleton data are present from the quest2's native hand tracking nad given to OpenXR.  

[mbucchia]

Displaying the hands joins is really the easy part, faking the buttons and controller interactions is the hard part. It can't be done smoothly and give you a good experience. It requires the game to be designed specifically for hand interaction. Even with all the work I put into this mode for MSFS, it's still a pretty mediocre experience. I do not have time to invest into it for DCS.

I believe you can probably persist and try making it work. My guess is you need to change the interaction_profile in the config to something else, probably the oculus/touch_controller one. There are instructions on the website to do that.

[ZerXen]

2 minutes ago, mbucchia said:

Displaying the hands joins is really the easy part, faking the buttons and controller interactions is the hard part. It can't be done smoothly and give you a good experience. It requires the game to be designed specifically for hand interaction. Even with all the work I put into this mode for MSFS, it's still a pretty mediocre experience. I do not have time to invest into it for DCS.

I believe you can probably persist and try making it work. My guess is you need to change the interaction_profile in the config to something else, probably the oculus/touch_controller one. There are instructions on the website to do that.

For the first argument, I expected that DCS would know what to do with the hand skeleton, since they already are using hand skeletons for the leap motion implementation. So just feeding the OpenXR skeleton into the leap-motion one should do the trick if ED woud be willing. 

For the second argument, I tried all four interaction profiles available, I simply made sure basic two gestures of wrist "Squeeze" and "Pinch" are registering properly using the debug overlay (these gestures hitting "1.0" reliably) and then tried mapping nearly all /input/XXX combinations and made sure to name config in "DCS World.cfg" and stored in appdata openxr configs folder (verified openxr log if it was loaded). 

Simply no interaction profile / input combination was registered by DCS as controller emulation. But if I pickup a real controller, the DCS hand appears normally. 

[mbucchia]

13 minutes ago, ZerXen said:

So just feeding the OpenXR skeleton into the leap-motion one should do the trick if ED woud be willing. 

Yes this is correct, but 100% different from what OpenXR Toolkit is doing (which is feeding the hand joints data as motion controller).

It would be neat to write the tool you mentioned above, but it would be limited exclusively to DCS (and maybe a small handful of other games with Leap Motion support).

[ZerXen]

11 hours ago, mbucchia said:

Yes this is correct, but 100% different from what OpenXR Toolkit is doing (which is feeding the hand joints data as motion controller).

It would be neat to write the tool you mentioned above, but it would be limited exclusively to DCS (and maybe a small handful of other games with Leap Motion support).

Looking at the DCS's settings with leap motion, VRFree and some "haptic glove" options... looks like ED has multiple interfaces that all feed into the DCS's native hand skeletons. So instead of faking OpenXR as leap motion, it might be nice if simple ED makes OpenXR hands as additional input option for their native hands. Up to the wishlist section! 

And back to the OP topic, why no profile/input settings in OpenXR's controller emulation is being registered by DCS at all ?  I do not even see how to troubleshoot something like this.

[mbucchia]

Idk either. It used to work fine in DCS through OpenComposite. Probably just a badly emulated call that prevents DCS from seeing that the virtual controller is alive.

[ZerXen]

1 hour ago, mbucchia said:

Idk either. It used to work fine in DCS through OpenComposite. Probably just a badly emulated call that prevents DCS from seeing that the virtual controller is alive.

Do you think that is worth filing a bug report to OpenXR ? Or that is ED's problem? 

[mbucchia]

6 hours ago, ZerXen said:

Do you think that is worth filing a bug report to OpenXR ? Or that is ED's problem? 

Honestly it's 99% likely my bug. But as I said that feature in my program is pretty much abandoned and hasn't had development in a year.

[ZerXen]

49 minutes ago, mbucchia said:

Honestly it's 99% likely my bug. But as I said that feature in my program is pretty much abandoned and hasn't had development in a year.

If it was motion leap exclusive I would say that is a dead horse togehter with the whole product and ignore fixing this, but quest2 hand tracking is excelent, especially with the giant field of view the cameras have. If you can only fix the profiles/inputs so that DCS registers them, you would make my day and if more people with quest2 would know abou this, it might get popular. Now that OpenXR finally works with quest2 link, it also got performance increase to be more usable.  And as I said in OP as owner of both quest2 and G2, I would sacrifice G2 clarity/performance to get a good cockpit interaction.  Don't want to talk heresy but ever since VTOL VR I am spoiled and looking for good hotas/hands interaction combo for DCS, and this might be very close.  

[mbucchia]

I'll take a quick look next week BUT even if the interactions worked I am pretty sure you will not be having a good experience. But I guess we can't be sure until we try.

[ZerXen]

2 hours ago, mbucchia said:

I'll take a quick look next week BUT even if the interactions worked I am pretty sure you will not be having a good experience. But I guess we can't be sure until we try.

Honestly, I base my logic on this:
1) People in general do not have good experience with pure hand tracking (and by extension also leap motion) in DCS
2) Using a controller is in general pretty good experience clicking cockpit buttons (the drawback why people do not use it is to put it down/up every time to grab a hotas)

BUT, since I have one of these cheap finger bluetooth mouse things, but modded mine to kill the movement sensor) I would really like to try only using hand tracking for the 6DOF laser pointer, but I have a physical left/right/scroll on my hand. Might just be a great combination. 

Edited July 5, 2023 by ZerXen

[actually_fred]

16 hours ago, ZerXen said:

Using a controller is in general pretty good experience clicking cockpit buttons

I disagree with this

- moving your hand *near* a cockpit switch triggers it, and AFAIK this can't be turned off. This is especially problematic when using hand tracking and controller emulation rather than real controllers, especially with switched around the throttle: it's way too easy to accidentally turn off an A10C's fuel pumps when going to full throttle
- the bindings for oculus touch controllers (thumbstick directions rather than buttons) need precision to trigger reliably, and are unintuitive.

Edited July 6, 2023 by actually_fred

[ZerXen]

9 hours ago, actually_fred said:

I disagree with this

You got me there with DCS, I was actually thinking about VTOL VR here more, but keep with me here for a second.

VTOL VR has an interaction with cockpit instruments using a controller triggered only by controller's trigger button and no physical finger hitting a button (and it also has a super nice stickiness/magnetism of the visual hand to cockpit switches to compensate hand shaking).  But this is chicken and egg problem, since in DCS we need physical HOTAS, so we first need heavy proliferation of hand tracking (where cheap quest2 camera hand tracking comes in) and then we can make a nice video/wishlist for ED to rally around to get them to adopt a bit more VTOL VR like hand interactions, just based on hand tracking so that players can keep their hotas systems.  But here we are talking about native hands-skeleton in DCS, for that I can only do a wishlist item here: 

However from you as OpenXR guy, I would only like the emulated interface to work with DCS as a more universal work-around for now and I would love to try it, especially if it is just some small bug that blocks the profiles/inputs from the emilation to get registered by DCS. 

Edited July 6, 2023 by ZerXen

[Dentedend10]

On 7/5/2023 at 4:14 PM, actually_fred said:

I disagree with this

- moving your hand *near* a cockpit switch triggers it, and AFAIK this can't be turned off. This is especially problematic when using hand tracking and controller emulation rather than real controllers, especially with switched around the throttle: it's way too easy to accidentally turn off an A10C's fuel pumps when going to full throttle
- the bindings for oculus touch controllers (thumbstick directions rather than buttons) need precision to trigger reliably, and are unintuitive.

 

I was having the issue of accidental ejections in the hornet. Then I found that you can turn on an option in the settings where interaction only occurs as long as you are holding the grip button. So interaction is much more reliable that way. However, it also gets rid of the haptic feedback that happens when interacting with buttons, which is sad

[rafalito1]

En 5/7/2023 a las 9:05, mbucchia dijo:

Idk either. It used to work fine in DCS through OpenComposite. Probably just a badly emulated call that prevents DCS from seeing that the virtual controller is alive.

Yes, this guy gets the hand tracking to work perfectly in DCS using OpenComposite:

The trick is:
-DCS Updater/Launcher Utility to run DCS with SteamVR Client.
-OpenComposite to switch to OpenXR.
-OpenXR Toolkit to use hand tracking as usual.

I don't understand why it doesn't work with native OpenXR.

Please Mr. Mbucchia, many of virtual fighter pilot want VR+Hand Tracking in DCS...

[mbucchia]

52 minutes ago, rafalito1 said:

many of virtual fighter pilot want VR+Hand Tracking in DCS...

And the ONLY WAY to have it PROPERLY done is by having the game implementing it. So please redirect your requests to ED.

That hand-to-motion-controller hack is terrible, impossible to setup, very uncomfortable to use, and that's why I abandoned it over 18 months ago...

[Dogmanbird]

 

Mbucchia - I now have stable hand tracking everywhere in my headset's field of view and my experience with your toolkit's tracking in MSFS has been excellent, very close to being as stable as a VR controller. So far it's working just as I'm wishing for in DCS below. 

The leap "hand" tracking is working great for me through DCS' native openxr. I'm happy not having any "finger" interaction with switches or dials and can imagine the complexity around developing that. My wish is to be able to point with my hand using the laser (this can remain on all the time if necessary) and disable all other gestures so they don't accidentally interact with anything. I'm happy to flick switches with hotas buttons or the mouse buttons once the laser hits them.

Is it difficult to achieve this via DCS native Openxr without the ED team making changes?  

I'm aware of the fingers app pointctrl styled approach (i finally got it working) but it's not for me. I appreciate the authors time creating it and sharing it.

If using open composite via steam allows the emulation via openxr toolkit, I suppose I could disable all gestures except for the finger-gun, which so far has been the least likely to cause accidental activations in MSFS for me. That would do the job. 

Edited September 27, 2023 by Dogmanbird

[Dogmanbird]

On 9/15/2023 at 7:17 AM, rafalito1 said:

Yes, this guy gets the hand tracking to work perfectly in DCS using OpenComposite:

The trick is:
-DCS Updater/Launcher Utility to run DCS with SteamVR Client.
-OpenComposite to switch to OpenXR.
-OpenXR Toolkit to use hand tracking as usual.

I don't understand why it doesn't work with native OpenXR.

Please Mr. Mbucchia, many of virtual fighter pilot want VR+Hand Tracking in DCS...

Hi, have you managed to get this to work for you? Just need some clarification of whether it's only the native dcs openxr that's broken or if OpenComposite is too. Not quite sure based on the earlier posts.

I haven't been able to get the emulation option to appear in the toolkit menu. I'm using the steam install and no updater/launcher program

at this stage;

The leap api is running according to the toolkit companion app.

The system wide opencomposite has dcs set as openxr

steamvr has openxr set as the active runtime

steamvr has both the the leap api and openxr toolkit api turned on

openxr tool is reporting (has loaded?) the toolkit api layer and leap hand tracking api layer

I've tried starting dcs with and without the force steamVR within the steam launcher

Edited September 27, 2023 by Dogmanbird