Eye tracking control of in cockpit switches

[Mr_sukebe]

We now have several VR headsets that support eye tracking, i.e. Aero, HP Omni, Neo Pico Eye, Quest Pro and Pimax.

Following on from this thread:

It did make me wonder if it was viable to use the eye tracking to look at a switch, and then have a button on a HOTAS to control the equivalent of left/right buttons on a mouse, i.e. similar to the PointCntrl, but without the need for the device.

Any thoughts?

[AngleOff66]

Something like this perhaps?

https://www.reddit.com/r/hoggit/comments/10pd6hn/for_those_who_wanted_to_track_the_progress_of_my/

VR mouse-pointer-head-tracking utility

I think the OP mentioned looking for help with programming for Windows .NET/StereoKit/OpenXR/C#

I think that something like this would be great, and I would hope that it comes through.

Edit: has anyone tried camera mouse? http://www.cameramouse.org/

Edited July 21, 2023 by AngleOff66

[mbucchia]

The accuracy of the eye trackers in those devices is enough for coarse pointing, but in something as dense as in a virtual cockpit, you won't go very far, especially if the pointing isn't implemented in the game (where the game could snap onto game elements).

I haven't tried Quest Pro - but I think that's what you have? If so, if you download the latest version of my Quad-Views-Foveated (formerly Meta-Foveated), you can enable a debug option `debug_eye_gaze=1` in the settings that will show a purple dot where you are looking. You will be able to judge the accuracy and also the "stability" of pointing. Let me know what you think.

Doing it myself on Aero, Crystal and G2 Omnicept, I can tell that pointing isn't good enough for a crowded cockpit. You can stare at something, and the dot is always "close" but not quite there. So if you have a bunch of knobs very close to each other, you wouldn't be able to precisely pick one. You will also see how "shaky" the pointing is, the dot is always doing slight movements. It's too sensitive, though that part can maybe improved by filtering. 

For a solution like that to work out, you'd need a way to declutter the cockpit (less dense amount of controls) which would be less realistic. You would need the game to implement the pointing feature, so that it would "snap on" to the closest control. 

It would be great if a game developer tried, but I think there are many challenges before this can be an even remotely pleasant experience. 

Edited July 21, 2023 by mbucchia

[Mr_sukebe]

Have just added in the debug_eye_gaze=1 option.  I'll update with findings later today.

BTW, thanks again for the DFR.  That it seriously awesome. 

[slughead]

With the Quest Pro you have hand tracking and can control the cockpit instruments that way. Pinch with first finger and thumb for mouse click, second finger and thumb for other mouse click. Pinch and raise or lower your hand to turn dials.

[Mr_sukebe]

1 hour ago, slughead said:

With the Quest Pro you have hand tracking and can control the cockpit instruments that way. Pinch with first finger and thumb for mouse click, second finger and thumb for other mouse click. Pinch and raise or lower your hand to turn dials.

Tried it.  I thought that it was rubbish

[dburne]

2 hours ago, slughead said:

With the Quest Pro you have hand tracking and can control the cockpit instruments that way. Pinch with first finger and thumb for mouse click, second finger and thumb for other mouse click. Pinch and raise or lower your hand to turn dials.

They have that working now when running in PC-VR mode? I know at one time it was only in stand alone mode, but has been quite a long time since I ran an Oculus headset.

Being able to select a switch by just looking at it, I do not see how that could work at all. I would rather use a mouse. However I do have Point CTRL I use every day with DCS, and absolutely love it. It freed up a few buttons for me as well. Bad part is currently production is stalled with over a year wait, hopefully it will resume again soon. I waited right at a year for mine which I got early this year, and it was so worth the wait. I also am on the waiting list for a second unit as well, just to have if needed.

Edited July 21, 2023 by dburne

[Mr_sukebe]

mbucchia>

OK, just tested the use of the Debug Eye Gaze:

• Certainly does produce two fairly large purple squares, which I assume is one per eye
• If you drew a line between the two, the object I'm looking at is very much in the middle (horizontally), if slightly lower, which I'm guessing is down to calibration
• The really interesting part being that I tried looking at a variety of points in my Mi8 and it really did appear that there was a good chance that I could focus on what I was looking for

So it would "appear" to be a viable method to identify where you're looking. 

To make this work, I think it would probably need the following:

• To be able to display a crosshair, i.e. similar to what you see when you use a mouse in cockpit, with that crosshair between directly between the current two purple dots
• To be able to toggle the crosshair on and off with say a HOTAS button
• To be able to use other HOTAS buttons to equate to using the left/right mouse buttons

Thoughts?

[mbucchia]

Thanks for trying. Yes I should've mentioned that the purple thing is used for the foveated rendering, and instead for an "aim" thing we would be using the center. So you guessed right.

Doing the crosshair isn't hard, what is difficult is how to translate the input into something usable by the game. This is truly something the game NEEDS to implement themselves. Otherwise we have to resort to faking input...

- We can't really use the mouse. Let me explain why. All games have different ways of capturing mouse input, because mouse is an OS concept, and not an OpenXR one, and many of them do not follow best practices. For example games will typically use relative mouse movement upon blocking the cursor into a confined space. "Relative" movement means impossible for an external software to calibrate absolute position (and by nature your eye gaze is a point projected in absolute coordinates on a plane in front of you). This is why you can't see/use the mouse when your game window has focus (and you need to Alt-Tab or Win button to "release" the cursor to Windows). Also apps like DCS are a good example: if you look closely at DCS after Alt-Tabbing, you'll notice that the cursor on the desktop and the cursor inside DCS aren't aligned at all. This is due to the difference in absolute vs relative coordinates. It would be impossible for a "driver" to correlate these coordinate systems.

- One thing that OpenXR eye tracking and the game use in common is the motion controller. There is a direct and easy way to correlate your eye gaze position and your motion controller. But this isn't practical for use. We could make your eye gaze act as a virtual motion controller, but then it comes with many issues, such as your motion controller is not "stuck to your face", but instead at arm length. So we'd have to somehow fake a certain depth for the "eye-to-controller". The problem is we don't know what the scene in the game looks like, so we'd arbitrarily pick a depth, which sometimes will be too far from the cockpit switches, and sometimes will be behind the cockpit switches. That won't be usable.

- I hear there are those "pointCTRL" and @actually_fred has a software called HTCC to control it from OpenXR. I think it could in theory plumb the eye gaze input into it, but I suspect it would have the same issues described above related to motion controller usage.

So it's unclear where to go from there. Idk if you have any idea based on the currently available input methods in the game, and how we could fake the input using the eye tracker, now that I've explained some of the challenges with it.

[Hempstead]

Works just fine... 

[Mr_sukebe]

Thanks for both of the above responses and thoughts, very much appreciated.

I hadn't realised about the difference in control between the mouse and otherwise.  I'd like to ask ED, however there's really not going to be many potential users, and it feels like a truly inefficient use of their time.

I'll go have a ponder about potential solutions.  I'm very much NOT an SME in Windows and similar.

 

 

 

[slughead]

On 7/21/2023 at 4:43 PM, dburne said:

They have that working now when running in PC-VR mode? I know at one time it was only in stand alone mode, but has been quite a long time since I ran an Oculus headset.

Being able to select a switch by just looking at it, I do not see how that could work at all. I would rather use a mouse. However I do have Point CTRL I use every day with DCS, and absolutely love it. It freed up a few buttons for me as well. Bad part is currently production is stalled with over a year wait, hopefully it will resume again soon. I waited right at a year for mine which I got early this year, and it was so worth the wait. I also am on the waiting list for a second unit as well, just to have if needed.

 

Yes. You have to jump through a couple of hoops such as enabling developer mode, enabling hand-tracking passthrough (I think) and install Fred's HTCC (Hand tracked cockpit clicking) tool: fredemmott/HTCC: Hand tracking for flight simulator cockpit clicking (github.com)

[winghunter]

On 7/21/2023 at 10:43 PM, mbucchia said:

- We can't really use the mouse. Let me explain why. All games have different ways of capturing mouse input, because mouse is an OS concept, and not an OpenXR one, and many of them do not follow best practices. For example games will typically use relative mouse movement upon blocking the cursor into a confined space. "Relative" movement means impossible for an external software to calibrate absolute position (and by nature your eye gaze is a point projected in absolute coordinates on a plane in front of you). This is why you can't see/use the mouse when your game window has focus (and you need to Alt-Tab or Win button to "release" the cursor to Windows). Also apps like DCS are a good example: if you look closely at DCS after Alt-Tabbing, you'll notice that the cursor on the desktop and the cursor inside DCS aren't aligned at all. This is due to the difference in absolute vs relative coordinates. It would be impossible for a "driver" to correlate these coordinate systems.

 

Im using JoyToKey to map the thumbstick absolute position to the absolute mouse position. Something similar could be used to map x/y eye coordinates to mouse position.

Edited August 29, 2023 by winghunter

[Dentedend10]

On 7/23/2023 at 7:42 AM, slughead said:

Yes. You have to jump through a couple of hoops such as enabling developer mode, enabling hand-tracking passthrough (I think) and install Fred's HTCC (Hand tracked cockpit clicking) tool: fredemmott/HTCC: Hand tracking for flight simulator cockpit clicking (github.com)

Hi! Are you actively using it? I haven’t got around to set it up yet due to some bad experience with Leapmotion. I have the QPro so if it’s good I’d set it up

[winghunter]

got it to work

Edited August 29, 2023 by winghunter

[slughead]

1 hour ago, Dentedend10 said:

Hi! Are you actively using it? I haven’t got around to set it up yet due to some bad experience with Leapmotion. I have the QPro so if it’s good I’d set it up

Yes, I use HTCC all the time. Leap motion didn't work particularly well for me.

[TAIPAN_]

Have you guys tried out the "button as left mouse" joystick bind I the UI Layer category? 

It would remove one step of the process, though I'm not sure if it still relies on actual mouse cursor position which probably still needs to move to the center of the gaze. I'm still setting up DCS on my new computer or I'd try it myself, maybe later. 

[dburne]

1 minute ago, TAIPAN_ said:

Have you guys tried out the "button as left mouse" joystick bind I the UI Layer category? 

It would remove one step of the process, though I'm not sure if it still relies on actual mouse cursor position which probably still needs to move to the center of the gaze. I'm still setting up DCS on my new computer or I'd try it myself, maybe later. 

 

Sure I have assigned buttons to mouse clicks in the UI layer and used in the past. Handy with VR so do not have to fumble for mouse. This was before i got Point CTRL.

[Qcumber]

9 minutes ago, TAIPAN_ said:

Have you guys tried out the "button as left mouse" joystick bind I the UI Layer category?

Yes. I use this all the time but you need to disable the mouse. It's great for switches immediately in front of you but a pain when you want to access low left or low right. Necksafer helps a bit with this. 

[TAIPAN_]

On 8/31/2023 at 11:40 PM, Qcumber said:

Yes. I use this all the time but you need to disable the mouse. It's great for switches immediately in front of you but a pain when you want to access low left or low right. Necksafer helps a bit with this. 

Just tested this with the Pimax crystal, binding a joystick button as left click. 

Currently you do not have to disable the mouse. 

I push the joystick button, which makes the cursor appear, then I move my head over MFD buttons and push the joystick button it works quite well. 

Then I moved the mouse and it worked fine to take over, wasn't disabled at all. 

I think this is quite a nice combo for MFDs & UFCs etc where you don't need to rotate knobs and just need alot of button pushing while flying, saves you from taking the hand off the stick to find the mouse. 

Edited September 9, 2023 by TAIPAN_

[Qcumber]

10 minutes ago, TAIPAN_ said:

Then I moved the mouse and it worked fine to take over, wasn't disabled at all. 

What I find is that after a few minutes you need to recentre the cursor and then sometimes it does not react immediately. For me there seems to be an issue in transferring between head movements and using the mouse. 

[TAIPAN_]

1 hour ago, Qcumber said:

What I find is that after a few minutes you need to recentre the cursor and then sometimes it does not react immediately. For me there seems to be an issue in transferring between head movements and using the mouse. 

Yeah I think if you use the mouse it stays where you put it for a bit, but once it disappears it will center upon reappearing when you push the joystick button.

So Joystick button switching to mouse is OK.

Mouse switching to joystick button is not OK, unless you wait for the cursor to disappear/recentre first.

But I need more flying time to know for sure, haven't had time to do a full long mission with the new headset yet. Just been performance tuning and adjusting my default head positions in the settings at the moment.

[dburne]

4 hours ago, TAIPAN_ said:

Just tested this with the Pimax crystal, binding a joystick button as left click. 

Currently you do not have to disable the mouse. 

I push the joystick button, which makes the cursor appear, then I move my head over MFD buttons and push the joystick button it works quite well. 

Then I moved the mouse and it worked fine to take over, wasn't disabled at all. 

I think this is quite a nice combo for MFDs & UFCs etc where you don't need to rotate knobs and just need alot of button pushing while flying, saves you from taking the hand off the stick to find the mouse. 

 

 

Binding mouse buttons to joystick has always worked very well for me. I do not anymore since getting Point CTRL but had no real complaints using buttons.

[TAIPAN_]

14 hours ago, dburne said:

 

Binding mouse buttons to joystick has always worked very well for me. I do not anymore since getting Point CTRL but had no real complaints using buttons.

Great thanks, now the job of finding a spare hat switch that is not going to impact any particular aircraft.... 

Not sure if I will bind the scroll wheel as well, or just the 2 buttons.

[TAIPAN_]

Did some more flying with MFD planes, and the Hornet with the center MFD especially was a bit annoying for the neck.

The button I bind to show the cursor (bound to the LALT C command), I also added a voice attack bind to move it to the center and then down 100 pixels.

That gets around the issue of the cursor being where it was last time you used the mouse, and the 100 pixel offset helps your neck. Usually you won't be clicking up high anyway, so putting the cursor slightly below the gaze seems to work well.

So I'm using a Hat Depress for this, hat left for mouse left, right for mouse right, up and down for wheel up and down. I only wish the mouse wheel command would work on a HOLD down basis, as turning on the JHMCS etc means you have to spam it to take the brightness all the way up.

null