leapMotion implementation possibility

[havok2]

for the sake of completeness

 

Here you find my fix for the emulator + video. Have fun testing it.

https://forums.eagle.ru/showthread.php?t=275731

[Snake122]

Sorry I've been laying low on the thread, been setting up the Pimax 5k+ this week then downloaded 2.5.6 Stable most of the day. I have the Leap Motion attached to the Pimax. For a little bit I was getting the Leap Motion driver to work in the Steam VR house. Then I exited to restart everything so I could run 2.5.6 for the first time and in the restart of SteamVR made me loose some of the controller icons and now I just have the Pimax and Lighthouse icons and no Leap Motion tracking. If I run DCLeap it will detect the gestures and I can occasionally see the hands for less than a second. Not surprised since Pimax isn't supposed to support it but I'll see if there is anything else I can do.

 

The interesting thing is that inside the Pimax planet (think simple SteamVR room for Pimax) before SteamVR is running, I get hand tracking with the Leap Motion! I know they have agreed to work together to bring out a new version so maybe that is why, but it isn't usable as a controller as far as I can tell.

Edited May 30, 2020 by Snake122

[Snake122]

Okay, guys check me here now that I have a better understanding

 

Current DCS Leap Motion Implementation:

1-Basic SteamVR Hand tracking provided by Drivers by CBuchner, SDraw, and the latest by Havok2. Emulates Vive Controllers in SyeamVR. Havok2 may or may not have optimized for DCS, but these would in theory work with any SteamVR app (and headset).

2- DCLeap- Adds mouse emulation and hand gestures for DCS with the previously mentioned drivers

3- VRK- adds basic hand tracking for Oculus users (these users also have the option to use 1 and 2 if they use the --force_steam_vr command line parameter)

 

 

 

Pimax- I did get it to work with the SteamVR driver this morning. I had to run VRK kneeboard app and for some reason this activated the SteamVR Leap Motion driver that doesn't load if you enter DCS with just how PiTool activates SteamVR.

[havok2]

Okay, guys check me here now that I have a better understanding

 

Current DCS Leap Motion Implementation:

1-Basic SteamVR Hand tracking provided by Drivers by CBuchner, SDraw, and the latest by Havok2. Emulates Vive Controllers in SyeamVR. Havok2 may or may not have optimized for DCS, but these would in theory work with any SteamVR app (and headset).

Pimax- I did get it to work with the SteamVR driver this morning. I had to run VRK kneeboard app and for some reason this activated the SteamVR Leap Motion driver that doesn't load if you enter DCS with just how PiTool activates SteamVR.

 

Hello, Snake122,

I just fixed the offset. With the SDraw driver the hands are rotated by about 45°. The only "optimization" is that you now press and hold the A Button with the "gun gesture" and activate the hand to use it. This seemed to be more intuitive than pressing the thumb and middle finger together and triggering the buttons with the index finger.

A friend of mine was able to use his hands with the Pimax 5k with my variant. He used SteamVR for that.

 

Best regards.

[Snake122]

Hello, Snake122,

I just fixed the offset. With the SDraw driver the hands are rotated by about 45°. The only "optimization" is that you now press and hold the A Button with the "gun gesture" and activate the hand to use it. This seemed to be more intuitive than pressing the thumb and middle finger together and triggering the buttons with the index finger.

A friend of mine was able to use his hands with the Pimax 5k with my variant. He used SteamVR for that.

 

Best regards.

Now I understand your offset, I thought you were saying an x/y/z offset because you can control that in VRK. Does the middle finger+thumb pinch work with standard SD Draw? Or is that the only way to flip switches and why it has been hard for me since I've been trying how you describe you changed it too all along:lol: I read the DCLeap guide but not anything for the standard driver. I thought that was just added by DCLeap for turning knobs. Did you also implement the DCS VR palm up/palm down altogether or is it easier now without the offset?

Edited May 30, 2020 by Snake122

[havok2]

Hey

 

exactly I was thinking how great it is that there is an emulator and then the hands were completely far away from my real hand position. So it was absolutely useless. I corrected that.

 

Yes the gestures were actually quite well recognized. Here is the driver from me with fix, but without adaptation for DCS. So grabbing is not the same as pressing key A and grabbing. https://workupload.com/file/wZmDQPRq2YA

 

I had tried it with my Valve Index Controllers in the beginning - which is of course crazy with Hotas. There I had to hold button A pressed to activate the hand. Then you saw this "laser". If you now touch one of the buttons or switches in the cockpit with your virtual finger, a left mouse button click is executed. If you turn your hand around, no collision triggering worked. Here you had to use the trigger to press the right mouse button. I just haven't figured out how to pull the lift to fold the wings. Mouse wheel is not used. Because I often use the cockpit with the head I have one button for right click, one for left click and two for the mouse wheel. So I can go with my hand to the switch and use the mouse wheel on the joystick to pull the lever.

 

With CTRL+O / CTRL+P you can also disable or enable the left and right hand. I use most times only the left hand. Here are the complete controls for the Emulator: https://github.com/SDraw/driver_leap/blob/master/README.md#valve-index-controllers-emulation

 

Edited May 30, 2020 by havok2

[streakeagle]

Here you find my fix for the emulator + video. Have fun testing it.

https://forums.eagle.ru/showthread.php?t=275731

 

I will try it out.

 

If capto gloves were working correctly, I would have bought them... especially since they had the option to connect to the arms of the virtual pilot.

 

But Leap Motion (now known as Ultraleap) is working fairly well and more practical. But without haptic feedback to let you know when you are in proximity of a switch, it can be very dangerous. Accidently dropping the landing gear at high speed or other such fun inadvertent button/lever pushes.

 

<edit>These modified drivers work a lot better. I can see my hands in SteamVR and they work fairly well in DCS World. It definitely takes some practice to get used to the limitations of the field of view of the leap motion sensor and its ability to see your hands well enough when looking down and to the sides. I inadvertently was moving the radio volume knob on the MiG-21bis. It is on the right hand side near the switches I was turning on during startup. While experimenting with the knob trying to perfect setting it at minimum, mid-range, and maximum, I hit the canopy jettison control... oops! I quickly got pretty good at toggling switches, but pushing buttons that need to be held in for a few seconds is a little trickier. I got the MiG-21bis started up 100% using VR hands, got permission to takeoff and was just about to push the throttle forward when DCS crashed. Running SteamVR on top of Oculus with LeapMotion as well has having VoiceAttack/Vaicom Pro and SimShaker (vibrating seat) running in parallel is a bit much on the system. It seemed like it crashed when I glanced down. Leap motion doesn't seem to like my looking anywhere but mostly straight ahead and level, whereas lowering my head, especially to the sides where I have physical barriers, gives it trouble seeing and recognizing the hand position and shape.

Edited May 30, 2020 by streakeagle

[Snake122]

Hey

 

exactly I was thinking how great it is that there is an emulator and then the hands were completely far away from my real hand position. So it was absolutely useless. I corrected that.

 

Yes the gestures were actually quite well recognized. Here is the driver from me with fix, but without adaptation for DCS. So grabbing is not the same as pressing key A and grabbing.

 

I had tried it with my Valve Index Controllers in the beginning - which is of course crazy with Hotas. There I had to hold button A pressed to activate the hand. Then you saw this "laser". If you now touch one of the buttons or switches in the cockpit with your virtual finger, a left mouse button click is executed. If you turn your hand around, no collision triggering worked. Here you had to use the trigger to press the right mouse button. I just haven't figured out how to pull the lift to fold the wings. Mouse wheel is not used. Because I often use the cockpit with the head I have one button for right click, one for left click and two for the mouse wheel. So I can go with my hand to the switch and use the mouse wheel on the joystick to pull the lever.

 

With CTRL+O / CTRL+P you can also disable or enable the left and right hand. I use most times only the left hand. Here are the complete controls for the Emulator:

Got a chance to try it out today and really like the changes. Its cool that it makes them the Vive Index "Knuckles" instead of the old wands. I also didn't have to run VRK first nor the Leap Motion app, it just works with my Pimax. Great stuff, thanks!

 

I did notice that it added them as a controller in DCS as well. Since I'm new to SteamVR and 2.5.6, is this the driver or the the latest update that made are mappable (the only physical VR controllers I have used are Oculus and they look to be hardcoded)?

 

What's the best way to manipulate rotary knobs with your driver update?

 

I will try it out.

 

If capto gloves were working correctly, I would have bought them... especially since they had the option to connect to the arms of the virtual pilot.

 

But Leap Motion (now known as Ultraleap) is working fairly well and more practical. But without haptic feedback to let you know when you are in proximity of a switch, it can be very dangerous. Accidently dropping the landing gear at high speed or other such fun inadvertent button/lever pushes.

 

<edit>These modified drivers work a lot better. I can see my hands in SteamVR and they work fairly well in DCS World. It definitely takes some practice to get used to the limitations of the field of view of the leap motion sensor and its ability to see your hands well enough when looking down and to the sides. I inadvertently was moving the radio volume knob on the MiG-21bis. It is on the right hand side near the switches I was turning on during startup. While experimenting with the knob trying to perfect setting it at minimum, mid-range, and maximum, I hit the canopy jettison control... oops! I quickly got pretty good at toggling switches, but pushing buttons that need to be held in for a few seconds is a little trickier. I got the MiG-21bis started up 100% using VR hands, got permission to takeoff and was just about to push the throttle forward when DCS crashed. Running SteamVR on top of Oculus with LeapMotion as well has having VoiceAttack/Vaicom Pro and SimShaker (vibrating seat) running in parallel is a bit much on the system. It seemed like it crashed when I glanced down. Leap motion doesn't seem to like my looking anywhere but mostly straight ahead and level, whereas lowering my head, especially to the sides where I have physical barriers, gives it trouble seeing and recognizing the hand position and shape.

FWIW, I believe Ultraleap will still call the current unit the Leap Motion. I'm watching now for the Ultraleap+Pimax unit to be available. It is supposed to have better hardware and a better mounting solution.

 

I'm running all of those apps as plus VRK and DCS hasn't crashed on me. It definitely starts to lose some tracking down to the sides. I feel like my really low mount on my Pimax (literally level with my mouth) is not helping as well.

[havok2]

...What's the best way to manipulate rotary knobs with your driver update?...

 

Hey, Snake,

as I said I only adapted the driver/emulator. The control is as specified by DCS for the Valve Index Controller. I use it for knobs like I use it with the head controller. I use the switch and keep the "left mouse button" pressed. Now I can look left and right with my head and turn the knob. So with the emulator you have to do the gun gesture and then aim the switch. Now you trigger the knob with your index finger and move your hand to the left or right. Some switches like the DDI brightned always require a trigger with your index finger to turn the knob. Here the whole thing does not work. Some of them seem to only turn when you move the mouse wheel or a mapped button. But here the problem lies in the implementation by DCS. I think that it basically accesses the Capto Glove implementation. Try and error - if you have try it first with real valve index controllers.

[Kariyann71]

Version 1.20 of DCLeap now available, see the first post to learn about new features.

 

 

Kariyann

[streakeagle]

Version 1.20 of DCLeap now available, see the first post to learn about new features.

 

 

Kariyann

 

I downloaded and installed it, but it is too late for me to play with it tonight. I will try it out tomorrow and provide feedback.

[streakeagle]

I got overwhelmed by the new patch yesterday: P-47, channel terrain, and MiG-21 updates. I spent most of the night playing with the P-47. Maybe tonight I will have time to try this out again.

[streakeagle]

It is working a lot better than the last time I used it. Part of that might be the SteamVR driver modifications referenced in an earlier post that got the controllers/hands aligned more correctly. I have a lot of difficulty getting a response to pulling the trigger or pinching. When I show hands, I can see the correct gesture, but it won't "click"... but then while I am struggling to get it working, it will click... and on the wrong thing. It is partially a function of what area I am trying to work (left/center/right and high/center/low) and the size of the trigger areas for that particular switch/button/lever. Some controls I can operate perfectly and others I have to struggle long and hard to move the switch/turn the knob.

 

I still find the gloved VR hand controllers built in to DCS World VR settings a lot smoother and more intuitive. I can be much more precise and can easily tell whether my hands are being recognized correctly or not.

 

But neither one is reliable enough for me to use regularly. Too much time trying to move small switches/buttons that I can hit instantly using a mouse. Too many accidental switch/button presses while struggling to press the right one or while simply looking around while my hands are on the stick and throttle. I need something that doesn't lose track and clicks exactly where and when I want it to click. I had my VR hands gut stuck holding the stick. They wouldn't move the stick, but they wouldn't let go either. Not sure what I did, but I got my left hand stuck while unsticking my right hand. Some gesture/position got me stuck and some other gesture/position got me unstuck.

Edited June 4, 2020 by streakeagle

[Kariyann71]

It is working a lot better than the last time I used it. Part of that might be the SteamVR driver modifications referenced in an earlier post that got the controllers/hands aligned more correctly. I have a lot of difficulty getting a response to pulling the trigger or pinching. When I show hands, I can see the correct gesture, but it won't "click"... but then while I am struggling to get it working, it will click... and on the wrong thing. It is partially a function of what area I am trying to work (left/center/right and high/center/low) and the size of the trigger areas for that particular switch/button/lever. Some controls I can operate perfectly and others I have to struggle long and hard to move the switch/turn the knob.

 

I still find the gloved VR hand controllers built in to DCS World VR settings a lot smoother and more intuitive. I can be much more precise and can easily tell whether my hands are being recognized correctly or not.

 

But neither one is reliable enough for me to use regularly. Too much time trying to move small switches/buttons that I can hit instantly using a mouse. Too many accidental switch/button presses while struggling to press the right one or while simply looking around while my hands are on the stick and throttle. I need something that doesn't lose track and clicks exactly where and when I want it to click. I had my VR hands gut stuck holding the stick. They wouldn't move the stick, but they wouldn't let go either. Not sure what I did, but I got my left hand stuck while unsticking my right hand. Some gesture/position got me stuck and some other gesture/position got me unstuck.

 

 

Hello Streakeagle,

 

 

you can adjust pinch sensitivity in the main menu. Take a look at the last part of DCLeap Manual to see how pinch is working in DCLeap.

About trigger, DCLeap detect fingers state, so to perform a click you have to

 

- thumb extended

- index not extended

- middle either

- ring either

- pinky either

 

 

About the global cursor sensitivity, you can adjust each axe in the main menu. Cursor sensitivity is associated to your desktop screen resolution.

For example, mine is 1920*1080, and my DCLeap's cursor sentivities are set both axes to 0.75.

 

 

I will try this weekend the havoc's modified drivers.

[streakeagle]

Hello Streakeagle,

 

 

you can adjust pinch sensitivity in the main menu. Take a look at the last part of DCLeap Manual to see how pinch is working in DCLeap.

About trigger, DCLeap detect fingers state, so to perform a click you have to

 

- thumb extended

- index not extended

- middle either

- ring either

- pinky either

 

 

About the global cursor sensitivity, you can adjust each axe in the main menu. Cursor sensitivity is associated to your desktop screen resolution.

For example, mine is 1920*1080, and my DCLeap's cursor sentivities are set both axes to 0.75.

 

 

I will try this weekend the havoc's modified drivers.

 

I spent some time playing with the triggering settings and enabled DCLeap's hands to help verify what the camera is seeing. When I use my right hand to flip switches to the front or front left, I get fairly good behavior. If I try to go to the far left and or down, it is tricky to get the cameras to see the hand in a way that recognizes the trigger. The process of flipping my hand around for camera recognition causes inadvertent control problems: either flip the wrong way or bump into another control or moving too much to hit the intended control. Just clicking on menu buttons is challenging for me with DCLeap, whereas that is effortless for me using the in-game VR hands.

 

My desktop resolution is 4K. It seems like the resolution selected when starting DCLeap affects the mouse sensitivity. I use the highest one listed that matches my aspect ratio because 3840 x 2160 isn't on the list.

Edited June 6, 2020 by streakeagle

[Om]

So what do you use now while playing? a mouse?

[streakeagle]

So what do you use now while playing? a mouse?

 

Most of the key functions are on my HOTAS. I mainly use the mouse during startup.

 

But if the VR hands worked reliably, I would use them whenever practical.

 

I looked at the point control mouse emulator. I considered getting it, but after seeing the finger control size/shape and how it would affect using my stick... i.e. clearance on the trigger finger, I decided against it. Hand tracking is the way to go. It just needs to mature and be natively supported by the game and VR headset rather than having to use even more hardware and software. Oculus is heading that direction. I was hoping they would make it native to Rift S and that ED would support it. In the mean time, Leap Motion is a cool toy to pay with, but I won't use it for serious flying when I don't want a canopy inadvertently jettisoned or to crash into the ground while trying to flip a switch.

 

A good mix for me would be to have a home pit that replicates the virtual pit and just use the VR hands to know where your real hands are. That means you would still be pushing real buttons, which means no hand tracking/gesture accidents and no need for special gloves to provide haptic feedback. The problem with that approach is you need to buy or build a replica cockpit for each aircraft you fly, or at least have interchangeable panels with proper control arrangements for VR interaction.

[Snake122]

Most of the key functions are on my HOTAS. I mainly use the mouse during startup.

 

But if the VR hands worked reliably, I would use them whenever practical.

 

I looked at the point control mouse emulator. I considered getting it, but after seeing the finger control size/shape and how it would affect using my stick... i.e. clearance on the trigger finger, I decided against it. Hand tracking is the way to go. It just needs to mature and be natively supported by the game and VR headset rather than having to use even more hardware and software. Oculus is heading that direction. I was hoping they would make it native to Rift S and that ED would support it. In the mean time, Leap Motion is a cool toy to pay with, but I won't use it for serious flying when I don't want a canopy inadvertently jettisoned or to crash into the ground while trying to flip a switch.

 

A good mix for me would be to have a home pit that replicates the virtual pit and just use the VR hands to know where your real hands are. That means you would still be pushing real buttons, which means no hand tracking/gesture accidents and no need for special gloves to provide haptic feedback. The problem with that approach is you need to buy or build a replica cockpit for each aircraft you fly, or at least have interchangeable panels with proper control arrangements for VR interaction.

Honestly, I've been flying with the Leap Motion off because the hand tracking causes too much FPS loss with the Pimax.

 

Have you tried binding the hand on/off keystrokes? When I get back around to it I was just going to keep the right hand off unless I really needed it. Left hand maybe on or maybe just map both commands to the HOTAS so you don't get the VR move the stick at least.

 

As I mentioned, I am on the PointCtrl wait list and you should check out the new 2.0 controllers that will be coming out soon. They are reduced in size. It really does sound the closest like what you want. One problem with your good mix wish is that it will still be very hard to line up the switches in the physical cockpit with where they are in VR. I have to remember this everytime I reach for my MFDs since they aren't where they are in VR for all the planes that have them. If the switches in your physical setup don't line up exactly for every switch is in VR, you could be flipping the next one over. Plus you will have dead spaces due to HOTAS bases ect. One of the ironic pluses of PointCtrl is there is no Z axis so you don't have to do the laser gun of the VR controllers if you can't put your hand deep enough to flip the switch by the VR push.

 

There is an interesting video out there of a simple plywood panel without switches being used with PointCtrl. At first I didn't get why, but it stabilizes your hands to click things. You then get the visual confirmation on screen which switch you are activating and then the tactile feedback as you push the button. Not ideal for sure but probably the closest you are going to get to your dream.

 

Om,

 

I just received a Kensington trackball with scroll ring for my left hand that I'm going to try. Right hand can still reach my normal mouse. Again, Leap Motion and Pimax work but something has to give with performance. The Rift S controller support in DCS isn't bad persay, but having to put the controller down, use HOTAS and then pick it up, then flip switch isn't optimal either. But the mouse control definitely loses some of the immersion effect

 

 

 

I wonder if like in Vaicom voice command stages, a sound telling you you are on a switch and then a louder switch flipping sound would help Leap Motion since we can't get the tactile feed back that the VR controllers give with vibration...

[streakeagle]

being able to turn the hands on/off easily is a good thing. that is an advantage of DCSLeap. But if I am in VR, having to use keyboard presses to toggle them isn't going to cut it.

 

I would prefer to use the existing DCS virtual hands with on/off mappable to any desired control. I didn't even assess the FPS cost of using Leap Motion / SteamVR / DCSLeap. I will have to try that this week.

[streakeagle]

I didn't even start DCSLeap and saw a noticeable drop in fps. I have my settings tuned to try to hit 80 fps when possible and avoid drops below 40 fps. Without Leapmotion/SteamVR, I was hitting 80 fps in high altitude/clear sky and stable at 40 fps cruising low over the Persian Gulf while flying the F-86 instant action takeoff without SteamVR/LeapMotion. With SteamVR/LeapMotion, I saw steady 35 to 38 fps with drops to 30 fps... that is a mission with no combat or heavy load of AI.

 

I want this tech so bad, but the hardware/software needs to step up to the capability I want. I guess I should wait until VR itself runs as resolutions/frame rates/visual quality I like before loading down my PC with extra software. I didn't really expect a hit in this situation. If I have 4 cores and DCS World uses at most 2 cores, I thought the extra overhead would be sucked up by the other two cores. But DCS world isn't really well coded for multi-cpu and it still has to interface with everything else while trying to do what it normally does, so I guess it is just too much for my system at the moment.

 

I love the tech and it is fun to try it out, but it is going to collect dust over the short term.

[ashwad]

Wonderful, wonderful tool. I love actually giving a salute before launch. I love reaching for the gear lever and flaps switch with my own hands. No mouse. No controller!

 

This is great. Please keep up the great work.

[Kariyann71]

Wonderful, wonderful tool. I love actually giving a salute before launch. I love reaching for the gear lever and flaps switch with my own hands. No mouse. No controller!

 

This is great. Please keep up the great work.

 

 

Thank you for your kinds words.

[Harlikwin]

 

I looked at the point control mouse emulator. I considered getting it, but after seeing the finger control size/shape and how it would affect using my stick... i.e. clearance on the trigger finger, I decided against it.

 

What stick are you using? I have like 5 different sticks and 3 different throttles and use point control with all of them without issue. In fact I can't even imagine what the problem could be. You wear them on the first knuckle coming out of your hand, not the tip. It is the BEST solution for DCS currently.

[streakeagle]

My first knuckle tends to rest up against the top of stick grips MCG Pro and TM Hornet.

 

But back to the original topic of the leap motion controller, I had not used it in a while and I saw a check box in VR options for hide hands unless palm grip detected. So, I decided to try out leap motion again and see if the VR option would work with the current setup for Leap Motion / Steam VR. My hope was that the hand controller would appear when I made the pointing finger gesture, and disappear when I didn't have that gesture. But the checkbox option isn't connected to the finger pointing gesture. So the hands were still always on unless moved out of the camera view.

 

Within 30 seconds of trying to start an F-86, I remembered why I stopped messing with this. Besides the lower fps performance in DCS, I couldn't easily operate the engine start switch with a momentary down and a maintained up position. In the struggle to figure out how to get it to move the way it needed to: down then up, my canopy went flying. The right hand is the natural way to operate that switch. But the camera has a hard time seeing the hand at all, much less the correct gesture when I am down that low and to the right. I either have to crank my head around to try to give the camera a better view, and/or use my left hand to operate the switch correctly. It is just to slow and inaccurate for me.

 

I love the concept and in some cockpits it works fairly well, but it is unsuitable for me to use during combat missions, especially in multiplayer, where performance is already pushed to its limits and there is no margin for error or delays. Clearly the technology is very close to working the way it could and should, but the hardware and software could use a few improvements before it performs well enough to surpass other means of cockpit control operation in a VR environment. If the DCS World core was coded specifically to employ leap motion so it could operate the controls more easily/accurately without unintended results and wouldn't need to run Steam VR on top of Oculus, it could already be much more useful. I haven't tried leap motion in the FlyInside flight simulator (I own it but don't have it installed right now), but I understand leap motions is natively supported in that sim.

 

Leap Motion has another product that is very promising: Stratos. Hand tracking with haptic feedback. Ultrasound is used to let you know when you are touching a control. DCS World would have to support it natively, but imagine reaching out into the air and "feeling" the button, switch, or knob you are operating. The future is so bright, I gotta wear shades :matrix: But will this stuff be useful in my lifetime? I have waited decades for VR headsets to reach consumer distribution levels and even now VR is still "early adopter" level. I want to enter the VR world have have it be so enjoyable, stable, and immersive that it is hard to turn to if off.

Edited August 3, 2020 by streakeagle

[elmaui]

Hi guys, I'm using Reverb G2 with LeapMotion, installed GitHub DriveLeap but it seems won't work with DCS.

Anyone still using LeapMotion with DCS in January 2021 and get it work properly?

Thanks!

[Finally works but is really too slow...]

Edited January 24, 2021 by elmaui