Jump to content

Fingers - Leap hand tracking and physical switches for VR and others


Recommended Posts

Try as I might, I haven't been able to get a PointCTRL, and the various Leap/VR Controller integrations were difficult to set up and fiddly in use (no tactile feedback, lots of accidental activation of switches). So I made my own solution involving a Leap for hand tracking and dedicated finger-rings for button pressing. CPU usage and framerate impact is negligible, and it's a single executable, with no driver changes or SteamVR config required.

 

Like PointCTRL, the Leap input translates into DCS mouse input rather than VR input, here's an older video of what it's like in use:
 

 

The setup is a head-mounted Leap (though with some modification, you could fixed-mount it for monitor setups), the Fingers software, and some custom-made five-direction button rings (the directions enable left click/right click/scroll up/scroll down/disable).

 

The rings are micro-USB rechargeable, and can handle six-eight hour sessions, with two weeks of standby time.


image.pngimage.png

image.pngimage.png

 

You can use it with other ring-worn button inputs, like the Kokuyo presenter, though they may interfere with hand tracking more.

 

I'm sharing this as a hobby project for discussion and feedback - if you want a robust solution you can buy, you will likely have better luck with PointCTRL, as I imagine its an easier plug and play solution, with tracking probably more reliable than the Leap's, particularly in extremes (twisting around to switches further back).

 

PS the rings also double as great mute buttons and controls for Zoom/Google Meet 🙂

 


Edited by frenzon
  • Like 15
  • Thanks 2
Link to comment
Share on other sites

  • frenzon changed the title to Fingers - Leap hand tracking and physical switches for VR and others
  • 3 months later...

Interesting. I have PCTRL but with some new leap motion integration in DCS ive grabbed a leap motion controller too. This seems to have a more stable pointer than PCTRL so it would be fun to try it.  Maybe i can use the pointctrl as a input device instead of making a new ring too.  PCTRL has the same issue with things in the back and close being somewhat tricky to click. 

Mind sharing the HW details on your ring? I already have the switches needed. I really prefer to be able to set thing as i'd like and DIY instead of relying on other people. 

 

Cheers!


Edited by trigen

1080 ti, i7700k 5ghz, 16gb 3600 cl14 ddr4 oc

Link to comment
Share on other sites

  • 2 months later...

Updated to work with the Ultraleap v5.2 (Gemini) tracking software:
https://github.com/glenmurphy/fingers/releases/tag/v0.9.3

Would appreciate any tests/feedback - the beta of the software had a 'stops tracking after 30m' issue that I haven't seen here, but who knows.

On the rings side, the global chip shortage has meant that I need to redesign the rings around a different module, and now that I live in Australia, getting PCBs/modules delivered is adding a lot of iteration time 😕

Link to comment
Share on other sites

23 hours ago, Chuls said:

hi @frenzon,

this looks very interesting to me but I'm not sure how it would be possible to test this. The repo says it requires a custom bluetooth ring but you haven't published that yet, right?
is it possible to check something?

The ring is just for doing mouse button presses, so if you have those mapped to HOTAS controls it will work without the rings (nowhere near as good)

Just to update people on the rings - the chip shortage meant that I had to go back to square one on the ring design as the modules I was depending on are out of stock for at least a year (and probably forever), so I'm now awaiting some ublox BMD-350 modules.

Link to comment
Share on other sites

16 hours ago, frenzon said:

The ring is just for doing mouse button presses, so if you have those mapped to HOTAS controls it will work without the rings (nowhere near as good)

How do you detect if the click was made from left or right ring?

 

Edit: No big explanation needed, it's enough to tell me in which file and method to look 😉 


Edited by Copprhead
Link to comment
Share on other sites

Nice solution. If the Leap could generate mouse clicks through finger movement then we could do away with any additional hardware. For example, point with index finger, left click with thumb touch to first segment of second finger, right click with thumb touch to second segment of second finger. It seems to know where all the segments of the finger are so maybe this is possible. Then no extra hardware is needed.

i9-13900k (stock), ASUS TUF 4090 OC, Gigabyte Z790 UD AX, Corsair Vengeance Black 64GB 5200MHz DDR5, 1 x 1TB WD_BLACK SN850x GEN4 NVMe, 1 x 2TB WD_BLACK SN850x GEN4 NVMe, 1 x SanDisk SSD PLUS 1 TB SATA III 2.5 Inch Internal SSD, Thermaltake Toughpower GF3 1650W PCIe GEN 5.0 ATX3.0 PSU, Corsair AIO iLink cooler, Corsair Graphite 780t Full Tower ATX Case (White), Meta Quest Pro VR headset, Samsung Odyssey Neo G9 49" UltraWide Dual Quad HD 240Hz Curved Gaming Monitor, Thrustmaster Viper Mission Pack, Thrustmaster Cougar F-16 throttle, Thrustmaster Warthog throttle and stick, Thrustmaster F-18 stick, Thrustmaster TPR rudder, Winwing ICP, Edifer e25HD Luna HD speakers. 1 Gbps uplink and downlink FTTP. Windows 11. Hot tub, hot wife. Living the dream!

Link to comment
Share on other sites

@frenzonThanks for this good Sir, it looks freaking amazing!
Can you elaborate more where you get those rings or which parts they consist of? Your version looks far superior to the bulky presenter solution. I'd buy some off you as well if that works for you. I just can't believe that I live in an age where this is finally all possible! 

| i9 12900K |  64GB DDR5-6000 | STRIX RTX 4090 OC | LG 38GN950 38" |

| Hanns-G HT225HPB | TIR 5 & Varjo Aero | Virpil Throttle & Stick | TM TPRs |

You don't stop playing because you grow old, you grow old because you stop playing.

Link to comment
Share on other sites

10 hours ago, slughead said:

Nice solution. If the Leap could generate mouse clicks through finger movement then we could do away with any additional hardware. For example, point with index finger, left click with thumb touch to first segment of second finger, right click with thumb touch to second segment of second finger. It seems to know where all the segments of the finger are so maybe this is possible. Then no extra hardware is needed.

I spent some time experimenting with this before, but it just wasn't reliable enough - either it introduced slight delays in clicking, moved the position of the hand as you were clicking, or just didn't register - in combination with the DCS cockpits having very tiny buttons, it meant that you'd frequently press the wrong thing. However, all of that was done before the Ultraleap Gemini update, so maybe I will give it another test! Thanks for reminding me.

1 hour ago, mhe said:

@frenzonThanks for this good Sir, it looks freaking amazing!
Can you elaborate more where you get those rings or which parts they consist of? Your version looks far superior to the bulky presenter solution. I'd buy some off you as well if that works for you. I just can't believe that I live in an age where this is finally all possible! 

I designed and made the rings - they're basically an NRF52832 module running Espruino, a five-way switch, and a tiny battery+usb port. I don't have any to sell right now because the world ran out of the modules I was using (Raytac MDBT42V), so I need to redesign them with the Ublox BMD-350.


Edited by frenzon
  • Like 2
Link to comment
Share on other sites

  • 2 months later...

@frenzon, Just wanted to say I just found this utility after struggling with the current (native) Leap support AND sitting on the PointCTRL waitlist for the better part of a year.  I've crafted my own rudamentary finger-buttons and, combined with your tool I literally could not ask for a better solution.  Thank you for this!

Link to comment
Share on other sites

17 minutes ago, Sielu said:

@frenzon, Just wanted to say I just found this utility after struggling with the current (native) Leap support AND sitting on the PointCTRL waitlist for the better part of a year.  I've crafted my own rudamentary finger-buttons and, combined with your tool I literally could not ask for a better solution.  Thank you for this!

Thank you for the feedback! What did you end up doing for the finger buttons?

As a small update, attached are pictures of the latest iteration of the rings - most of the work now is in figuring out the magnetic charging system.

PXL_20211213_034110951.jpg

PXL_20211213_034225943.jpg


Edited by frenzon
updating image sizes
  • Like 1
Link to comment
Share on other sites

3 minutes ago, frenzon said:

Thank you for the feedback! What did you end up doing for the finger buttons?

 

 

Lol, my solution is ... not nearly as elegant: just two sets of these hard wired to an Arduino mimicking an Xbox controller, which is then bound to right and left mouse in DCS.  The wires are annoying, but at least I don't have to worry about charging batteries?

98B1EABD-B1C0-4A24-976A-0E3E94FC239C.jpeg

  • Like 3
Link to comment
Share on other sites

  • 2 weeks later...

Anyone has an idea why the Fingers software closes 2 seconds after I start it? Is there a specific place I need to install it or what could be the problem?

Never mind about that. Reinstalled windows and it works perfectly. Amazing work, you actually made leap motion usable in DCS. Now all I need is the rings...


Edited by Bojevnik
Link to comment
Share on other sites

I'd like to volunteer my time and money on testing the rings as soon as they're available

Nice job

*8k Pimax with Leap ctrl here waiting

skunk160 | Win10 PRO 64bit | i7-4770K 3.50 GHz | 32GB DDR3/1866MHz | GIGABYTE GeForce GTX 1080 x2 | Oculus Rift S | Virpil MongoosT-50CM2 | Virpil F-14B grip | Virpil 200m Curved Extension | PointCTRL | Delta Sim TM Slew | Sim Bandit AHCP | MFG Crosswind Pedals | //FOX2 Switch Boxes | RECARO SPG Seat | AuraSound AST-2B-4 Pro Bass Transducer x2

//FOXTWO Multi-Role Combat Pit Build http://forums.eagle.ru/showthread.php?t=134745

Link to comment
Share on other sites

So I've upgraded my previous hard-wired arduino buttons to wireless versions using the same Espruino Module that @frenzon is using... although I'm using the breakout board, and my solution is, again... significantly less elegant than Frenzon's:

 

PkiSeQa.jpg

 

Couple decisions were made:

1. Didn't want to use LiPo's. I almost burned down an apartment a few years back with one of those (totally my fault, but... left a bad taste in my mouth).  And, while I know these modules use almost no power, I hate the idea of going to start up the sim and then having to wait for my hands to charge, so I wanted something hot-swappable.

2. I'm not good at soldering, so I opted to use the breakout board, which necessitated moving the 'brains' of the unit to the back of the hand, which then meant I needed a mounting solution so I bought some "fishing gloves" on Amazon and hot-glued 3D Printed mounts to those.  This has worked out surprisingly well, as I can actually rip them off and reposition them if needed.  Not that I've had to do that...

3. With the added real-estate, I was able to fit bigger buttons on.  I've got right and left click, as well as scroll up and down mapped to each finger pad.  I've got the Espruino board mimicking a mouse directly, so the system doesn't know it's not me clicking. 

With all that, I've still got great range of motion, and the buttons are out of the way of the controls!  Not to mention I feel like I've got Alyx's gravity gloves!

3ifTsYR.gif  63WQ9jz.gif  8VQQsaS.gif

Also surprising--and I did significant testing--Leap still manages to recognize my hands quite well.  Tracking is only an issue at really weird angles, like trying to get back-panel switches.  The other issues is that the Espruino boards tend to 'sleep' on their own to save power, which means that I sometimes have to click a bunch of buttons to get them to 'wake up' and start being mice again.  Annoying, but workable. 


Edited by Sielu
Link to comment
Share on other sites

On 1/17/2022 at 3:54 PM, Bojevnik said:

Anyone has an idea why the Fingers software closes 2 seconds after I start it? Is there a specific place I need to install it or what could be the problem?

Never mind about that. Reinstalled windows and it works perfectly. Amazing work, you actually made leap motion usable in DCS. Now all I need is the rings...

 

Any other options other than the nuclear option above to fix the disappearing?  run as admin and not same result

skunk160 | Win10 PRO 64bit | i7-4770K 3.50 GHz | 32GB DDR3/1866MHz | GIGABYTE GeForce GTX 1080 x2 | Oculus Rift S | Virpil MongoosT-50CM2 | Virpil F-14B grip | Virpil 200m Curved Extension | PointCTRL | Delta Sim TM Slew | Sim Bandit AHCP | MFG Crosswind Pedals | //FOX2 Switch Boxes | RECARO SPG Seat | AuraSound AST-2B-4 Pro Bass Transducer x2

//FOXTWO Multi-Role Combat Pit Build http://forums.eagle.ru/showthread.php?t=134745

Link to comment
Share on other sites

58 minutes ago, skunk160 said:

Any other options other than the nuclear option above to fix the disappearing?  run as admin and not same result

Interesting.  Runs perfectly for me, on two separate PCs.  One with LEAP service installed and one without.  Are you running W11 or W10?

Link to comment
Share on other sites

 

On 1/3/2022 at 10:13 PM, Sielu said:

Lol, my solution is ... not nearly as elegant: just two sets of these hard wired to an Arduino mimicking an Xbox controller, which is then bound to right and left mouse in DCS.  The wires are annoying, but at least I don't have to worry about charging batteries?

98B1EABD-B1C0-4A24-976A-0E3E94FC239C.jpeg

If possible could you make a short tutorial on how to do this? Which components did you use and how to connect the wires...


Edited by Bojevnik
Link to comment
Share on other sites

7 hours ago, skunk160 said:

Any other options other than the nuclear option above to fix the disappearing?  run as admin and not same result

Ah sorry you're experiencing this - I don't have a good idea of what it could be, but as a guess, do you have Bluetooth disabled on your system? Fingers might have a dependency on Bluetooth (which is how the rings work), so it might be crashing if it's not there.

Link to comment
Share on other sites

5 hours ago, frenzon said:

Ah sorry you're experiencing this - I don't have a good idea of what it could be, but as a guess, do you have Bluetooth disabled on your system? Fingers might have a dependency on Bluetooth (which is how the rings work), so it might be crashing if it's not there.

Hey, so if I disable bluetooth on my PC (switch it to Airplane mode) I get the exact behavior -- fingers closes after a few seconds.  

Link to comment
Share on other sites

10 hours ago, frenzon said:

Ah sorry you're experiencing this - I don't have a good idea of what it could be, but as a guess, do you have Bluetooth disabled on your system? Fingers might have a dependency on Bluetooth (which is how the rings work), so it might be crashing if it's not there.

ill check its probly off, i dont have BT rings 😕

skunk160 | Win10 PRO 64bit | i7-4770K 3.50 GHz | 32GB DDR3/1866MHz | GIGABYTE GeForce GTX 1080 x2 | Oculus Rift S | Virpil MongoosT-50CM2 | Virpil F-14B grip | Virpil 200m Curved Extension | PointCTRL | Delta Sim TM Slew | Sim Bandit AHCP | MFG Crosswind Pedals | //FOX2 Switch Boxes | RECARO SPG Seat | AuraSound AST-2B-4 Pro Bass Transducer x2

//FOXTWO Multi-Role Combat Pit Build http://forums.eagle.ru/showthread.php?t=134745

Link to comment
Share on other sites

17 hours ago, Bojevnik said:

 

If possible could you make a short tutorial on how to do this? Which components did you use and how to connect the wires...

 

Hey @Bojevnik, had to wait until the end of the day before I could finish a write-up.  The process is relatively simple but a was a little involved for me with the resources I had on-hand.  I only had an Arduino Uno when I started, which doesn't natively allow for connection to a PC as a controller (a 'Human Interface Device', or HID).  Therefore I had to use the UnoJoy library, with instructions posted on their github to get the Arduino working: https://github.com/AlanChatham/UnoJoy

I would recommend if you're starting from scratch, to use a different model of Arduino.  Later reivsions can connect to a PC natively as an HID, which makes the process a bit simpler.  A quick google search tells me that this library is probably acceptable: https://github.com/NicoHood/HID, and it looks like you could use an Arduino UNO here as well.  I'd recommend an Arduino Leonardo or Pro Micro to start with, for simplicity.  Then just follow the instructions in the libraries above to set them up as a game controller.  You don't need to worry about axes unless you want to.  I just mapped 4 buttons to mine, and left the rest of the available controller unassigned. 

The wiring is pretty simple.  I chose basic 12mm momentary buttons (https://www.amazon.com/TWTADE-Momentary-Tactile-Button-Switch/dp/B07CG6HVY9/ref=sr_1_48?crid=1LNA1PJU0ZESX&keywords=12mm+button&qid=1642627045&sprefix=12mm+button%2Caps%2C46&sr=8-48), which have a really nice click and are easy enough to wire.  When the button is clicked, it closes a circuit between the two halves of the button, so to wire you just connect one half to an output pin on the Arduino, and the other half to a ground (GND) pin on the Arduino.  You can share multiple GND pins between multiple buttons, but the 'source' pins have to be different.  When the button is clicked and the circuit closes, it "drives the pin low" or causes the voltage reading on the pin to drop, since it's now sending its electricity to ground.  The software reads this as a click and sends the appropriate report to the PC, where it's interpreted as a controller button click.  

I know that's a high-level runthrough, I hope it helps! 

  • Like 4
Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...