Jump to content

Fingers - Leap hand tracking and physical switches for VR and others


Recommended Posts

Bluetooth on works like a charm!!!  Rings ordered

skunk160 | Win10 PRO 64bit | i7-4770K 3.50 GHz | 32GB DDR3/1866MHz | GIGABYTE GeForce GTX 1080 x2 | Oculus Rift S | Virpil MongoosT-50CM2 | Virpil F-14B grip | Virpil 200m Curved Extension | PointCTRL | Delta Sim TM Slew | Sim Bandit AHCP | MFG Crosswind Pedals | //FOX2 Switch Boxes | RECARO SPG Seat | AuraSound AST-2B-4 Pro Bass Transducer x2

//FOXTWO Multi-Role Combat Pit Build http://forums.eagle.ru/showthread.php?t=134745

Link to comment
Share on other sites

23 hours ago, Sielu said:

Hey @Bojevnik, had to wait until the end of the day before I could finish a write-up.  The process is relatively simple but a was a little involved for me with the resources I had on-hand.  I only had an Arduino Uno when I started, which doesn't natively allow for connection to a PC as a controller (a 'Human Interface Device', or HID).  Therefore I had to use the UnoJoy library, with instructions posted on their github to get the Arduino working: https://github.com/AlanChatham/UnoJoy

I would recommend if you're starting from scratch, to use a different model of Arduino.  Later reivsions can connect to a PC natively as an HID, which makes the process a bit simpler.  A quick google search tells me that this library is probably acceptable: https://github.com/NicoHood/HID, and it looks like you could use an Arduino UNO here as well.  I'd recommend an Arduino Leonardo or Pro Micro to start with, for simplicity.  Then just follow the instructions in the libraries above to set them up as a game controller.  You don't need to worry about axes unless you want to.  I just mapped 4 buttons to mine, and left the rest of the available controller unassigned. 

The wiring is pretty simple.  I chose basic 12mm momentary buttons (https://www.amazon.com/TWTADE-Momentary-Tactile-Button-Switch/dp/B07CG6HVY9/ref=sr_1_48?crid=1LNA1PJU0ZESX&keywords=12mm+button&qid=1642627045&sprefix=12mm+button%2Caps%2C46&sr=8-48), which have a really nice click and are easy enough to wire.  When the button is clicked, it closes a circuit between the two halves of the button, so to wire you just connect one half to an output pin on the Arduino, and the other half to a ground (GND) pin on the Arduino.  You can share multiple GND pins between multiple buttons, but the 'source' pins have to be different.  When the button is clicked and the circuit closes, it "drives the pin low" or causes the voltage reading on the pin to drop, since it's now sending its electricity to ground.  The software reads this as a click and sends the appropriate report to the PC, where it's interpreted as a controller button click.  

I know that's a high-level runthrough, I hope it helps! 

Thanks for the info, much appreciated. 👍

Link to comment
Share on other sites

Hey folks, made a quick video showing off fingers tracking + my wireless button gloves.  Doing a cold start on the F16.  Hope this inspires folks?

 Sorry the video is blurry in spots... camera kept having a mind of its own on focus.

  • Like 5
  • Thanks 1
Link to comment
Share on other sites

  • 2 weeks later...
13 hours ago, A.F. said:

how do you take notes in the video? (min 1:25)
You are in VR... How can you see the notes under the HMD?

 

I'm using VRK and a basic wacom tablet strapped to my knee.

VRK App:

 

Wacom Tablet: https://www.wacom.com/en-us/products/pen-tablets/one-by-wacom

 

VRK took a couple evenings to get configured but now that it's rolling it's been great!

  • Like 2
Link to comment
Share on other sites

  • 4 months later...

I designed a case, that could also house a small LiPo battery. I have attached the stl files if anyone is interested.

I have not been able to get it to work though. I was able to flash the devices, and they are recognised as mice after restarting them. But I cant get them to register any mice clicks. I have no idea what I am doing wrong. I am running FingersApp.exe also.

 

 

IMG_20220612_173229_144.jpg

IMG_20220612_173252_342.jpg

IMG_20220612_173334_263.jpg

Left Finger.stl Left Finger battery box.stl


Edited by Tehrawk
  • Like 2
  • Thanks 2
Link to comment
Share on other sites

On 6/12/2022 at 12:41 PM, Tehrawk said:

I have not been able to get it to work though. I was able to flash the devices, and they are recognised as mice after restarting them. But I cant get them to register any mice clicks. I have no idea what I am doing wrong. I am running FingersApp.exe also.

Just a note - Tehrawk was able to fix their issue and I have updated the guidance on the RoosterHands github to hopefully prevent further mishaps!

  • Like 3
Link to comment
Share on other sites

Thanks again for your help Sielu.

I am stuck again though. I am unable to get my hands to register accurately in VR. I can get it to sort of work on Leap version 5.6.1.0. But the pointers are way off to the right of the screen, and they seem to drop connection and stop moving momentarily.

I have tried a few different versions. Including 4.1., 5.0 and 5.2. But none of those work at all. What software version should I be using. Do I need to copy over LeapC.dll from that version?

Link to comment
Share on other sites

21 minutes ago, Tehrawk said:

I am stuck again though. I am unable to get my hands to register accurately in VR. I can get it to sort of work on Leap version 5.6.1.0. But the pointers are way off to the right of the screen, and they seem to drop connection and stop moving momentarily.

I have tried a few different versions. Including 4.1., 5.0 and 5.2. But none of those work at all. What software version should I be using. Do I need to copy over LeapC.dll from that version?

Copying LeapC over can help. One thing to try is to use the Leap visualizer software (IIRC it's installed as "Tracking Vizualizer") and see if the hands register cleanly in there with or without your rings on (which look awesome btw). The visualizer might point to other sources of interference too.

  • Like 2
Link to comment
Share on other sites

I think I got it to work now. Everything looks fine in visualizer.

One thing that was throwing me off, was the right hand disappearing when the stick is not centered.

The other thing was having to disable and reenable Leap every time I start DCS.

What is the optimal way to use this? Hands invisible and no pointers?

Link to comment
Share on other sites

 

On 6/12/2022 at 12:41 PM, Tehrawk said:

I designed a case, that could also house a small LiPo battery. I have attached the stl files if anyone is interested.

I have not been able to get it to work though. I was able to flash the devices, and they are recognised as mice after restarting them. But I cant get them to register any mice clicks. I have no idea what I am doing wrong. I am running FingersApp.exe also.

 

 

IMG_20220612_173229_144.jpg

IMG_20220612_173252_342.jpg

IMG_20220612_173334_263.jpg

Left Finger.stl 169.32 kB · 2 downloads Left Finger battery box.stl 52.04 kB · 1 download

 


Nice button design for the mouse wheel. I am moving to a similar design for the next iteration of PointCTRL. 
Good Stuff!! I love seeing this type of home grown innovation.

Miles

  • Like 1
  • Thanks 1

Now shipping up to website Pre-Order Form date 2022/03/10

Pre-Order  Form Submission https://pointctrl.com/preorder-form/

PointCTRL Support Discord https://discord.gg/jH5FktJ

PintCTRL Website https://pointctrl.com/

PointCTRLsmall.jpg

Link to comment
Share on other sites

10 hours ago, Tehrawk said:

I think I got it to work now. Everything looks fine in visualizer.

One thing that was throwing me off, was the right hand disappearing when the stick is not centered.

The other thing was having to disable and reenable Leap every time I start DCS.

What is the optimal way to use this? Hands invisible and no pointers?

If you are using the Fingers app, you should disable all the Leap stuff in DCS - if you use them both at the same time things will go awry (because Fingers moves the mouse, and DCS does its own thing)


Edited by frenzon
  • Like 1
Link to comment
Share on other sites

20 hours ago, frenzon said:

If you are using the Fingers app, you should disable all the Leap stuff in DCS - if you use them both at the same time things will go awry (because Fingers moves the mouse, and DCS does its own thing)

 

Fantastic, thanks frenzon!

Link to comment
Share on other sites

  • 3 weeks later...

Hi,

@frenzonnice work with Fingers App! Thanks for Your effort! 🙂


I took my Leap out of the shelf, where it had been rotting for years, printed the mount for my Reverb G2, installed latest Gemini, installed your Fingers 0.9.3, enabled BlueTooth (I do not use it, it was also rotting since ordovik on my PC)  and viola! It works! 🙂

Unfortunately with ugly offset to right, i.e. mouse pointer is shifted 1/4 screen size to right and therefore it is very difficult to use mouse pointer in DCS as it is very close to right hand edge of visible area, see the screenshot,  😞  Mouse pointer is cca at red point, as PrintScreen does not include it. This right offset is present for both hands.

 

Is there any setting or calibration of Fingers app?
I have found user.config in
\\AppData\Local\FingersApp\FingersApp.exe_Url_oiz1xmo2tljpyqkdzivdgwc423rzymoi\1.0.0.0\

but there is nothing to meaningfully setup.

 

image.png


Edited by wju

i7-7820X@4.5Ghz; GPU 3080Ti, 64GB RAM, HP Reverb G2; VKB Ultimate stick; VIRPIL Throttle; Slaw Pedals; custom AX&BTN BOX

 

Link to comment
Share on other sites

well, I have found the root of my problem - the second monitor. When switched off, then FingersApp works as expected. When is Fingers running, it even does not mind when I switch second monitor back on. Mouse cursor behaves as expected.
Now its the time to build a mouse (wired version at first ) based on an Arduino. 🙂


Edited by wju
  • Like 1

i7-7820X@4.5Ghz; GPU 3080Ti, 64GB RAM, HP Reverb G2; VKB Ultimate stick; VIRPIL Throttle; Slaw Pedals; custom AX&BTN BOX

 

Link to comment
Share on other sites

On 2/8/2022 at 6:23 PM, Sielu said:

Hey folks, I've got a new version of my gloves printed.  Switched to smaller batteries and mounted them directly on the back of the hands. Much more comfortable!  Also, made a GitHub repo for the stl files and Espruino code I wrote:

https://github.com/sielu-rooster/roosterhands

PXL_20220209_000556303.MP.jpg

Saw a guy a long time ago take gloves and a Wii controller and camera and make a point control glove out of it. Worked great.....

 

Sempre Fortis

Link to comment
Share on other sites

On 7/4/2022 at 7:39 AM, wju said:

Hi,

@frenzonnice work with Fingers App! Thanks for Your effort! 🙂


I took my Leap out of the shelf, where it had been rotting for years, printed the mount for my Reverb G2, installed latest Gemini, installed your Fingers 0.9.3, enabled BlueTooth (I do not use it, it was also rotting since ordovik on my PC)  and viola! It works! 🙂

Unfortunately with ugly offset to right, i.e. mouse pointer is shifted 1/4 screen size to right and therefore it is very difficult to use mouse pointer in DCS as it is very close to right hand edge of visible area, see the screenshot,  😞  Mouse pointer is cca at red point, as PrintScreen does not include it. This right offset is present for both hands.

 

Is there any setting or calibration of Fingers app?
I have found user.config in
\\AppData\Local\FingersApp\FingersApp.exe_Url_oiz1xmo2tljpyqkdzivdgwc423rzymoi\1.0.0.0\

but there is nothing to meaningfully setup.

 

image.png

 

I've also had an issue with the latest version of Gemini and second monitors messing fingers up.  WMR Was creating 'extra' fake monitors to handle 'legacy' windows apps every time it started, which was really chewing up Fingers.  I could get it to work by re-starting fingers AFTER WMR, but disabling the extra monitors setting in WMR fixed it for good. 

 

16 hours ago, 352nd_Hoss said:

Saw a guy a long time ago take gloves and a Wii controller and camera and make a point control glove out of it. Worked great.....

 

I was considering something like that for a future revision.  Right now this whole solution doesn't work for beans in a certain other popular, WMR-based, non-combat flight simulator because it's cursor implementation doesn't like the winput() commands that Fingers sends to windows.  Outside of going down the rabbit hole of programming new OpenXR controllers entirely, I'm thinking of some way to get the Espruino's to send the PC a mouse change signal to kludge in support for other sims.  Just gotta figure out a way for the Espruino's to know where they are in space.

Link to comment
Share on other sites

1 hour ago, Sielu said:

Outside of going down the rabbit hole of programming new OpenXR controllers entirely

You don't have to implement it; OpenXR Toolkit already has a mapping from hands to controller.

I just got my leap motion thinking that the toolkit would deliver the vision you described. It appears that the OpenComposite project doesn't read controllers from the OpenXR layers.

Link to comment
Share on other sites

1 hour ago, j5bit said:

You don't have to implement it; OpenXR Toolkit already has a mapping from hands to controller.

I just got my leap motion thinking that the toolkit would deliver the vision you described. It appears that the OpenComposite project doesn't read controllers from the OpenXR layers.

I've played around with this functionality and have so far found it to be pretty lacking, so you're not missing much.  The toolkit solution emulates standard VR controllers based on hand position, orientation, and finger movements.  Needless to say, this gets real finicky real fast.  I actually find DCS's native leap implementation to be better, but both approaches are still weak due to two fundamental issues:

1. 3D vs 2D: with all the current hand/controller to plane interfaces there's a default requirement that your hand be in the 3D vicinity of the control you want to interact with.  This seems great in theory, but in reality we all have 'stuff' in the way that can prevent that from actually happening.  Thus, most solutions allow for a ray cast of some form from your hand to the control that you want, but this typically requires contorting your hand to 'point' to a given control, which (for me at least) feels SUPER unintuitive and not immersive.

2. Finger movement to buttons: Since basically every solution is emulating a standard VR controller, registering buttons is done via finger movement/wrist turning of some form.  This is problematic, especially for ray cast interaction, since typically moving your finger or wrist shifts you hand position slightly, creating a lot of 'missed' interactions.  Not to mention LEAP is far from infallible, and will frequently not catch the movement to trigger the interaction.

For these reasons, I like the Fingers solution better.  It is effectively a 'ray cast' style interaction, but the 'ray' is being cast from your head, not your hand.  You're effectively given two 'control points' for the ray, your hand AND your head, instead of just your hand which makes it more stable.  Then, relying on physical buttons to trigger the different interactions makes the interactions themselves more reliable and more stable.

My ideal solution would be to program a head-based ray cast through each identified hand from the LEAP, then use the Espruino buttons to register button interactions.  I'm not that good at programming (yet).

Link to comment
Share on other sites

8 hours ago, Sielu said:

I've also had an issue with the latest version of Gemini and second monitors messing fingers up.  WMR Was creating 'extra' fake monitors to handle 'legacy' windows apps every time it started, which was really chewing up Fingers.  I could get it to work by re-starting fingers AFTER WMR, but disabling the extra monitors setting in WMR fixed it for good.

meaning the "HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Holographic" trick or something else?

wired mouse version based on Arduino works fine, but the wire is annoying, maybe I am "the most ugly solution award" winner 🙂

anyway, espruinos ordered.

 

 

FM2.jpg

  • Like 1

i7-7820X@4.5Ghz; GPU 3080Ti, 64GB RAM, HP Reverb G2; VKB Ultimate stick; VIRPIL Throttle; Slaw Pedals; custom AX&BTN BOX

 

Link to comment
Share on other sites

1 hour ago, wju said:

meaning the "HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Holographic" trick or something else?

Nope, this option in WMR settings:

WMR Desktop Duplication.png

1 hour ago, wju said:

wired mouse version based on Arduino works fine, but the wire is annoying, maybe I am "the most ugly solution award" winner 🙂

anyway, espruinos ordered.

 

 

FM2.jpg

LMFAO, I love it.  And, yeah yours might actually beat my first button + hot glue monstrosity. 

Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...