Hand Tracking w/ Meta Quest 3

[Dogmanbird]

oh?  I don't think my system / headset has updated to it yet.

edit - still on v63 for both

Edited April 8 by Dogmanbird

[dureiken]

Hi

 

I just bought a Q3 and tried handtracking. I would like to use it to help me clicking on right button on a real ICP and MFD.

It seems to work pretty well but I have to extra open my index to have it straight in the game, is it the same for you ?

 

Thanks a lot

[Dogmanbird]

you can use the hand offset settings in the DCS options and see if that helps. I'm using the htcc app which doesn't require pointing with the finger (like shooting a laser beam from tip of finger towards button), instead it requires moving my arm/hand all around the cockpit area like I'm drawing on a giant tablet

The hands gets in the way of my vision, so I offset the left hand so it sits slightly to the right of the crosshair, and do the opposite with the right hand (which I rarely use) to keep them out of the way. All I'm concerned about is being able to see the crosshair over the button I need to operate.

I wish there was a way to adjust the opacity of the hands (or does it already exist?)

 

Edited April 8 by Dogmanbird

[Qcumber]

6 hours ago, Dogmanbird said:

I wish there was a way to adjust the opacity of the hands (or does it already exist?)

There is a guide on the HTCC page to make the hands invisible. 

On 4/4/2024 at 1:12 PM, Qcumber said:

or moves the opposite direction to my head movement and the hand tracking is overlayed on this. This makes it unusable. Has anyone else had this issue? 

Has anyone else seen this effect. My head tracking moves in the opposite direction when HTCC is enabled making it impossible to use. 

[Dogmanbird]

Thanks Qcumber, if it's still the same method as months back, it was scaling down the crosshair as well as the hands. Does that happen for you?

 

does turning off the headset beta updates option allow you to roll back to v63 to see if that helps?

I don't recall ever experiencing that. if you are using it cabled, when in oculus link, does the pinch gesture for turning on menu bar or recentering headset follow your hands correctly? (oculus logo in circle that follows hands)

Edited April 8 by Dogmanbird

[Qcumber]

3 minutes ago, Dogmanbird said:

Thanks Qcumber, if it's still the same method as months back, it was scaling down the crosshair as well as the hands. Does that happen for you?

I need to check. 

3 minutes ago, Dogmanbird said:

does turning off the headset beta updates option allow you to roll back to v63 to see if that helps? 

That could be the issue. I might give it a go. 

[Qcumber]

On 4/8/2024 at 8:12 AM, Qcumber said:

Has anyone else seen this effect. My head tracking moves in the opposite direction when HTCC is enabled making it impossible to use. 

I am still getting this. Has anyone else had this issue? Really I just want to be able to disable head tracking cursor but that's not an option in DCS.

[slughead]

On 4/4/2024 at 1:12 PM, Qcumber said:

Thanks. My settings are pretty much as yours. I tried matching them exactly and still no joy. Then I tried VD instead of oculus and it work! Something must have changed with the v64 version. 

Edit: it works but I have a conflict with head tracking. The cursor moves the opposite direction to my head movement and the hand tracking is overlayed on this. This makes it unusable. Has anyone else had this issue? 

 

If you move your head and keep your hand still, it will appear as though your hand is moving in the opposite direction to your head. That's because the hand is actually moving across the camera sensor. It's a little annoying because in VR but not DCS (so in the headset home space) the hands do not move as the head moves. I guess the headset tracks hand movement in relation to the background. I expect Meta doesn't export that environment data for PCVR use.

I've learnt to keep my head still while looking at the control I need to adjust.

What finger mouse are you using?

Oh, Meta's famous v63 release broke HTCC. More on this with a workaround here:

 

[Qcumber]

12 minutes ago, slughead said:

If you move your head and keep your hand still, it will appear as though your hand is moving in the opposite direction to your head. That's because the hand is actually moving across the camera sensor. It's a little annoying because in VR but not DCS (so in the headset home space) the hands do not move as the head moves. I guess the headset tracks hand movement in relation to the background. I expect Meta doesn't export that environment data for PCVR use.

I've learnt to keep my head still while looking at the control I need to adjust.

What finger mouse are you using?

Oh, Meta's famous v63 release broke HTCC. More on this with a workaround here:

 

Thanks. Yes. I followed your advice regarding v63 as I previously could only get HTCC to work with VD. I'm now running v64 and Fred's solution works for this too. 

I am using a Bluetooth finger mouse I bought from Amazon which works pretty well with head tracking/mouse. It does also work with HTCC in the cockpit if I keep my head still. The inverted head movement is worse in the menus making them almost impossible to use even if I keep my head still. 

[slughead]

19 minutes ago, Qcumber said:

Thanks. Yes. I followed your advice regarding v63 as I previously could only get HTCC to work with VD. I'm now running v64 and Fred's solution works for this too. 

I am using a Bluetooth finger mouse I bought from Amazon which works pretty well with head tracking/mouse. It does also work with HTCC in the cockpit if I keep my head still. The inverted head movement is worse in the menus making them almost impossible to use even if I keep my head still. 

Yeah, in menus, it's no good. I use a normal mouse for the menus prelaunch. In the cockpit, I have the menu controls mapped to my HOTAS. Were you aware you can move up, down, return, and select using joystick buttons in in-game menus?

I have developed my own finger mouse. I call it the Slugmouse. I have one for each hand.

Edited April 11 by slughead

[Qcumber]

Just now, slughead said:

In the cockpit, I have the menu controls mapped to my HOTAS

That's a good idea. I might try that. 

1 minute ago, slughead said:

I have developed my own finger mouse. I call it the Slugmouse. I have one for each hand.null

Yes. I've been watching the progress. How does it compare to a normal finger mouse? 

[slughead]

Just now, Qcumber said:

That's a good idea. I might try that. 

Yes. I've been watching the progress. How does it compare to a normal finger mouse? 

I haven't tried a regular finger mouse, so I'm unsure how to compare them. This one is compact, has five buttons, is Bluetooth, sleeps after 15 minutes of no use, wakes instantly on middle button press, charges via USB-C from 0 to 100% in under an hour, and has a battery life that lasts for days. In your experience, how would you say that compares?

[Qcumber]

7 minutes ago, slughead said:

I haven't tried a regular finger mouse, so I'm unsure how to compare them. This one is compact, has five buttons, is Bluetooth, sleeps after 15 minutes of no use, wakes instantly on middle button press, charges via USB-C from 0 to 100% in under an hour, and has a battery life that lasts for days. In your experience, how would you say that compares?

Much more function than a regular finger mouse! Have you considered a scroll wheel? I quite like the wheel for cockpit dials. 

[slughead]

Just now, Qcumber said:

Much more function than a regular finger mouse! Have you considered a scroll wheel? I quite like the wheel for cockpit dials. 

A scroll wheel would be difficult. However, it does have scroll up and down functionality with acceleration. So it has normal left and right mouse button emulation and scroll up and down. The scroll up and down is a single click per press, or if you press and hold, it scrolls faster, like spinning a scroll wheel. Then, there is the centre button mapped to the back button on one hand and the other to the middle mouse button. The scroll acceleration makes using the HSI and course controls, HMCS intensity, console dials, etc, much easier to use. You can also use right-click hold and move your hand up and down with HTCC to spin dials. Plus the firmware can be updated.

[andyc]

guys so i have hand tracking working in dcs with 0.62 and 0.63. However it seems to have stopped completely in 0.64 - tried turning dev mode off and on, both in the pc app and iphone app but dcs doesn't come up with a trackers option. Anyone else found this ? I am thinking this will need a ticket on the meta support forums but experience with them breaking open xr in 0.63 doesn't fill me with much hope.

 

[slughead]

1 minute ago, andyc said:

guys so i have hand tracking working in dcs with 0.62 and 0.63. However it seems to have stopped completely in 0.64 - tried turning dev mode off and on, both in the pc app and iphone app but dcs doesn't come up with a trackers option. Anyone else found this ? I am thinking this will need a ticket on the meta support forums but experience with them breaking open xr in 0.63 doesn't fill me with much hope.

 

You can find the solution here:

 

[andyc]

On 4/17/2024 at 8:00 PM, slughead said:

You can find the solution here:

 

yup that fixed it. thanks very much

[TheDogBadger]

Gents,

I am attempting to use handtracking. It works, well to a point, the hands are so shakey. Does anyone know if ED plans on doing any work on fixing that?

[Dogmanbird]

Hi, what type of tracking do you have, as in a Quest 3 only or a Leap Motion type device attached to Q3?

The Quest tracking is not so good. I find it too jittery for relying on the finger to move the crosshairs. You can use some other software to link the crosshairs to the wrist/palm rather than the finger, which will provide a more stable result, but you'll need to move your arm about (which in a way is more realistic I guess)

If it's a leap device, I get my best results by removing any unnecessary equipment from around the cockpit area that will reflect the IR light, or cover it with material that is non-ir reflective. IR spotlights fired down from above over your playing area also noticeably improves the Leap devices tracking. This gets the finger linked to crosshair tracking working very well. However, as good at is is, I still find flicking switches / dials too cumbersome this way with DCS as it currently is.

[TheDogBadger]

@Dogmanbird I use a Quest 3. As mentioned earlier, it is working, it is just that the hand shakes so much it is honestly not worth it at. Do you know if the leap motion hand tracking is better? What is the other software you mentioned? Honestly, I am torn between VR and just head tracking. Yes, VR is much more immersive, but at graphic quality cost. I have I I7 32gb 4070ti super so not the top contender gaming machine, but I would say above the middle.

[kablamoman]

The Q3 hand tracking actually works really well and is super easy to get setup using Virtual Desktop.

It quickly becomes apparent that the real issue is that the cockpit controls were not designed with this kind of fine finger tracking capability in mind.

Each module and maybe core DCS would have to be redesigned to take proper advantage of hand-tracked manipulation of cockpit controls, along the lines of what VTOL VR has done. I think there's a lot to be excited about as this kind of thing gets developed in the future (hopefully).

 

[slughead]

1 hour ago, TheDogBadger said:

@Dogmanbird I use a Quest 3. As mentioned earlier, it is working, it is just that the hand shakes so much it is honestly not worth it at. Do you know if the leap motion hand tracking is better? What is the other software you mentioned? Honestly, I am torn between VR and just head tracking. Yes, VR is much more immersive, but at graphic quality cost. I have I I7 32gb 4070ti super so not the top contender gaming machine, but I would say above the middle.

 

30 minutes ago, kablamoman said:

The Q3 hand tracking actually works really well and is super easy to get setup using Virtual Desktop.

It quickly becomes apparent that the real issue is that the cockpit controls were not designed with this kind of fine finger tracking capability in mind.

Each module and maybe core DCS would have to be redesigned to take proper advantage of hand-tracked manipulation of cockpit controls, along the lines of what VTOL VR has done. I think there's a lot to be excited about as this kind of thing gets developed in the future (hopefully).

 

If you're trying to control switches directly with your fingers as you would in real life, then nothing works well for this currently. I tried this and gave up quickly, so I put my order in for pointctrl, which tracks your hands (not fingers) and controls the mouse pointer position through where you position your hand. You then click buttons mounted to your fingers to perform mouse clicks.

However, almost two years later, I was still waiting for my order to be ready. So, I made my own devices to perform the mouse clicks using the Quest hand tracking and a piece of software called HTCC to position the mouse.

Here is a video of me doing a ramp start using two of the devices that I've dubbed the Slugmouse. Further details of this here:

 

[Dogmanbird]

On 5/8/2024 at 4:05 AM, TheDogBadger said:

@Dogmanbird I use a Quest 3. As mentioned earlier, it is working, it is just that the hand shakes so much it is honestly not worth it at. Do you know if the leap motion hand tracking is better? What is the other software you mentioned? Honestly, I am torn between VR and just head tracking. Yes, VR is much more immersive, but at graphic quality cost. I have I I7 32gb 4070ti super so not the top contender gaming machine, but I would say above the middle.

The leap motion v2 is significantly more stable than Q3 when used in fairly ideal conditions. It is negatively affected by reflections, nearby objects and the hands casting shadows, much more so than the Q3, even your clothing (trousers etc) can affect the tracking - so unfortunately if you want to get the best from it, it'll require stripping back a cockpit arrangement to just a seat, a throttle sitting on a pedestal, stick and pedals on the floor (no framework, physical mfds, monitor in front of you etc.), keep at least 1m in front of you clear of objects and some IR spotlights overhead (helps with quest 3 tracking too) with monitor off to the side and tilted downward slightly. I'd imagine this would be too much to ask of most people with their existing setup. However, this will give stable fingertip laser-pointing styled control over the crosshair throughout the cockpit, achievable with your hands close to your body at waist / chest height. I could rest my forearm on my body (keeping hand stable) and simply point a finger/laser at a switch with pretty good precision. Keeping the hand a little further away from body further improved it. 

The leap also adds some weight to the front of the headset. I already found the Q3 front heavy, though the halo strap does help. It also needs it's own USB connection to the PC. It was great on the G2 because the headset is quite light and i used it without the face plate. I made a pvc mount for it and 3m taped it to the front like in pic below. That mounting position seemed to provide the closest IRL hand v virtual hands alignment (without needing to adjust in game) and helped reduce the "hands moving in opposite direction to head" effect.

I haven't used the leap in several months as I'm somewhat happy to get by with the bare Q3 and the tablet style tracking linked to wrist, to keep things simple. I jump back and forth between flying and driving (a car setup to the right of the pic) and moving / disconnecting leap cable was a pain.

Definitely worth it if you have only the one setup, happy with a bare-bones cockpit and a little extra weight on the headset. I'd guess a lot of leap owners are using it in a regular cockpit arrangement with big monitor, keyboard and other hardware fairly close and likely not experiencing it at it's best.

 

nullnull

Edited May 10 by Dogmanbird

[TheDogBadger]

Thanks again, another question, is it just the HTCC app you use? I read somewhere about a mod that basically locks your cursor on a button/switch, I can't remember the name though.

Does the first version  of the leap work, or is it worth it to get v2? It must be on the dcs end as the quest 3 reason for it not to work as well as leap hand tracking. ED has added something within dcs for the leap to work it seems. 

[Dogmanbird]

imo it's not dcs creating the problem. I can see the difference in stability when viewing a pointed finger in the leap settings software versus oculus home. the leap is steady as a rock (under the right conditions). there is almost no lag with the leap, but there is some latency with the Q3, probably due to averaging / stabilising.

I use HTCC with the quest 3. i'm not using it in 'tablet mode' because the cursor/dot moves around to much when the headset moves. I'm using the other option.

There's another program called something like EZHands. I found that one posted on reddit but can't find anything on it lately. It's almost perfect for me but when your hands move towards the top of your fov, they start to jump up and down erratically. For some reason it's only at the top while sides and bottom are fine. The good thing with this one is that you can offset the cursor as desired - otherwise it's very similar to HTCC

I haven't used a leap v1, but from what I've read, they've improved the tracking stability and FOV with v2. V2 is also physically smaller and lighter.