Connecting 100000 (yes) buttons, switches & potis to a SINGLE USB interface for $30

[winghunter]

Hello

I'm in the planning stage of a simpit and what I already don't like is the hard wiring of components and having to use multiple arduinos in different spots.

I want the whole structure to be more modular as in being able to reassign and rewire buttons to different locations without having to build new software for the Arduinos.

So thinking of alternatives, here's my concept.

Theory

Can you think of a common computer input device that is capable of reading 1 million inputs 60 times a second ? Yes, an HD USB camera. These are typically able to send 720*1080 pixels of uncompressed RGB @ 60fps to the computer.

Now, for the camera to read our inputs we have to connect each switch directly to an LED, and position these LEDs in a grid. We could also simply buy a ready made LED matrix. That means in theory we could now read 1 million inputs per second, but in reality we probably need some spacing between camera pixels to increase resilience. Hence, if we leave 9 pixels around each input pixel we could still read 100000 inputs @60 times a second.

Mapping

To match each pixel to a control input is fairly easy, this can be done in a raspberry PI or as a program running on your computer.

Multi-stage switches can be mapped to a single LED using 200ø resistors in series, therefore controlling the LED intensity in multiple steps-

Potentiometers can be wired to an RGB LED controlling a color shift from red to blue or green which can be detected by the camera. For example we could build the last 2 lines with RGP LEDs which can be wired to potentiometer inputs.

I.e.

• LED 1,1 -> button 1 ( RED LED )
• LED 2,1 -> button 2 ( RED LED )
• LED 4,1 -> multi-stage rotation switch ( RED intensity )
• LED 1,40 -> potentiometer input

 

EDIT: On the software side you could group LEDs into "panels" and assign key combinations

 

Pros & Cons

The nice thing about having this LED matrix is that it works STANDALONE - without a computer, chip or software. It's just pure electronics and allows you to debug the system visually. The matrix would need to be enclosed in a box to not affect the camera with other light sources.

You also don't need ANY programming skills. You could simply use my software and assign each pixel to a button with a point & click interface.

The other nice thing is that this is not limited to DCS. Your hardware can work with any other flight sim, while the software emulates keystrokes and keeps your mappings.

Now, to wire up 100s buttons to LEDs is certainly no easy feat. But im wondering if it's really harder than wiring up 100s of switches to arduinos - and writing custom software for each in DCS bios. Also dealing with and updating multiple arduinos is not very maintainable.

A disadvantage would be that our input rate is now limited to 60 fps. So this input method may work well for switches or potis. But maybe it's not fast enough for your HOT firing trigger. Yet, i can only think of the firing trigger that requires really fast millisecond response times, any other input in my plane is totally fine at 60fps.

Another disadvantage may be that we loose some accuracy with potentiometers. Due to converting them to RGP light and reading it through a camera. For things like flaps this may not be an issue, but if you use a poti to move a cursor on a map this could be more twitchy.

 

 Skipping the LED step 

An optimisation would be to skip the LED step entirely. How is a $30 USB camera able to serialise this much data and send it to the computer in a manageable format ? If we can somehow connect directly to the camera's serialiser we can save some work and gain more precision.

 

Any thoughts?

Edited December 14, 2021 by winghunter

[No1sonuk]

Looks like a dynamic QR code system.

Do you really need that many inputs?

[winghunter]

Its a dual cockpit with RIO so I need a lot. But also I want the flexibility to add more things later, make it modular swap modules etc. This would be much easier with this setup as its very simple.  With this many inputs I can just swap entire sections without having to reprogram anything.

[gwheeler]

Now I'm imagining a system where the buttons, knobs, and switches have no electrical connections AT ALL but merely change some visible property on the underside of the panel, which is seen by a camera underneath the console pointing upward. For example: Rotating a knob topside would cause a marker at the other end of the shaft to rotate and the camera captures the precise position of the control to within a degree or so. You could swap out modular sections, which could be hot-pluggable because each one could have a QR code or some such in the corner to cue the system as to which one was installed, which would map to a profile describing the inputs and their locations. Of course, it's probably possible to do this from above as well, with a CV solution that simply looks at the actual panel state the same way you see it. That would introduce a bit more latency since it would be doing more image processing in a less controlled space but man, the possibilities...

 

 

[Gruman]

Interesting concept. Definitly outside the box. Looks like a fun little project which should be tested on the smallest scale possible... because I doubt you would save any amount of work and it will take a huge amount of time to fix all issues. And I doubt all those LED's, Wiring, Connectors, Camera, Case, Power Supplies, will be anywhere near your 30$ price tag.  

You don't like Arduinos? Dont use them. Grab any microcontroller, make a small PCB, decide on a BUS system and do your own modular system.

[yogi149]

Hi,

vor 10 Stunden schrieb winghunter:

Its a dual cockpit with RIO so I need a lot. But also I want the flexibility to add more things later, make it modular swap modules etc. This would be much easier with this setup as its very simple.  With this many inputs I can just swap entire sections without having to reprogram anything.

are you sure, that DCS can handle that amount of input you try to realise?

And can you tell me, how this unusual concept can minimize the wiring work?

You have to connect your sections to the LED cluster, as before to any BBI (button-box-interface). Than you have to make an interpreter for the LED-cluster image, which would need a standalone CPU-unit.

And you have to attach every interpreted input to a DCS function by selecting it from "one" input device.

If you change a section, how can you identify the changed inputs?

Nice idea, but I think that the work to get this running is much higher, than to stack some USB BBI and connect them to DCS.

 

 

But with this idea I finally know why WOPR (wargames) had so many lights.

[hsssonic]

Hi,

Nice thinking model...

But maybe you should take a look into camera codecs and their compression algorithms.
I don´t know any webcam that sends RAW Data via USB to the computer.

I don´t want to say it´s not possible, but you should take something like that into consideration.

 

[winghunter]

2 hours ago, hsssonic said:

But maybe you should take a look into camera codecs and their compression algorithms.
I don´t know any webcam that sends RAW Data via USB to the computer.

Only the expensive cameras do on device compression, the cheap ones send RAW data. These typically don't have a microprocessor capable of realtime encoding.

Than you have to make an interpreter for the LED-cluster image, which would need a standalone CPU-unit.

Yes, though I have written similar software to interpret realtime camera footage, it's easier than it sounds. The CPU load is also fairly low on todays systems, a raspberry PI can perfectly do this task which is around the same price tag as an Arduino.

Edited December 14, 2021 by winghunter

[winghunter]

5 hours ago, Gruman said:

Interesting concept. Definitly outside the box. Looks like a fun little project which should be tested on the smallest scale possible... because I doubt you would save any amount of work and it will take a huge amount of time to fix all issues. And I doubt all those LED's, Wiring, Connectors, Camera, Case, Power Supplies, will be anywhere near your 30$ price tag.  

You don't like Arduinos? Dont use them. Grab any microcontroller, make a small PCB, decide on a BUS system and do your own modular system.

I love Arduinos, but I hate dealing with 10 units and keeping them up to date. LED's are inexpensive, costing as little as 0.001$ (RED) to 0.01$ (RGB) per piece

https://www.alibaba.com/product-detail/5mm-LED-Diode-Dip-Light-Emitting_1600238959281.html

You would only need a single power supply or even a battery can do the job. The amount of wiring needed is comparable to wiring the Arduinos so there is no advantage here. The only advantage would be that you don't have to deal with different software for each arduino. Ideally you wont have to write any software as you could simply use my clicky software with a GUI to assign buttons to keys. The software would be open source and MIT licensed, for everyone to use.

Edited December 14, 2021 by winghunter
LED link

[winghunter]

8 hours ago, gwheeler said:

Now I'm imagining a system where the buttons, knobs, and switches have no electrical connections AT ALL but merely change some visible property on the underside of the panel, which is seen by a camera underneath the console pointing upward. For example: Rotating a knob topside would cause a marker at the other end of the shaft to rotate and the camera captures the precise position of the control to within a degree or so. You could swap out modular sections, which could be hot-pluggable because each one could have a QR code or some such in the corner to cue the system as to which one was installed, which would map to a profile describing the inputs and their locations. Of course, it's probably possible to do this from above as well, with a CV solution that simply looks at the actual panel state the same way you see it. That would introduce a bit more latency since it would be doing more image processing in a less controlled space but man, the possibilities...

 

 

I thought about this approach as well. I've already written CV applications but there are some limitations to this. First of all, your hands and head can obstruct the buttons, especially when you operate them. But also CV takes more processing power.

EDIT: Re-read your post. Thats a great idea to have mechanical flags on the underside. But you would still need one camera per panel. So you will have to deal with multiple cameras. But therefore less wiring...

Could be interesting

Edited December 14, 2021 by winghunter

[winghunter]

5 hours ago, yogi149 said:

If you change a section, how can you identify the changed inputs?

 

 

I think this would just be a software problem.

A) you could dedicate an entire different section of the LED matrix for each of your panels, as this is only limited by the amount of LEDs you can buy, 

LED's are inexpensive, costing as little as 0.001$ (RED) to 0.01$ (RGB) per piece

B) You could define each panel by grouping LED's in the software. Then you could simply swap the keybindings for a certain panel.

Of course you should also connect each panel via i.e. a flatband connector to the LED panel. So you can also easily swap elements on the hardware side.

Edited December 14, 2021 by winghunter

[gwheeler]

2 hours ago, winghunter said:

I thought about this approach as well. I've already written CV applications but there are some limitations to this. First of all, your hands and head can obstruct the buttons, especially when you operate them. But also CV takes more processing power.

EDIT: Re-read your post. Thats a great idea to have mechanical flags on the underside. But you would still need one camera per panel. So you will have to deal with multiple cameras. But therefore less wiring...

Could be interesting

 

So definitely underneath then, where the lighting and other visual characteristics can be more uniform and tightly controlled to allow for faster image processing. If the panel ID is readable on the underside (QR code) it could reference a profile that specifies the panel dimensions (while the QR marker also functions as an origin point for the panel-local coordinate system.) That way you can correct for perspective issues caused by the camera being relatively close to the panel.  Prototyping new panel layouts could be really fast and cheap this way, because there's no wiring involved and you can experiment with component placement before producing a final version with more fancy and expensive materials like laser-engraved acrylic.

 

 

[yogi149]

Hi,

I may be a lousy pug now, but if you swap the "Gear" panel for a "Floats" panel, which now has 3 more switches, you have to swap the whole section in your interpreter. The effort 1. to wire the panel so that the appropriate LEDs light up and then 2. to adapt the interpreter so that the right section is also available in the simulator is now higher for me than connecting a single panel with a USB controller. Or am I wrong?

I still don't see the minor effort to wire the switches for the LEDs appropriately instead of using inputs directly.
Apart from that, given the number of LEDs provided, the LED image is also a very high expense for the circuit board and the power supply.
Troubleshooting should be relatively difficult.

[winghunter]

Well if you used ready made LED panels they could get expensive and they also often require a driver software.

I thought about DIY LED panels using just a drilled board or 3D printed frame, where you stick in $0.001 LEDs. Then you would just leave enough space / empty LEDs for any module to be swapped without having to reconfigure 3 additional inputs.

 

Edited Sunday at 05:13 PM by winghunter

[winghunter]

1 hour ago, gwheeler said:

So definitely underneath then, where the lighting and other visual characteristics can be more uniform and tightly controlled to allow for faster image processing. If the panel ID is readable on the underside (QR code) it could reference a profile that specifies the panel dimensions (while the QR marker also functions as an origin point for the panel-local coordinate system.) That way you can correct for perspective issues caused by the camera being relatively close to the panel.  Prototyping new panel layouts could be really fast and cheap this way, because there's no wiring involved and you can experiment with component placement before producing a final version with more fancy and expensive materials like laser-engraved acrylic.

 

 

The idea of using QR codes on switches is growing on me. These could be 3D printed and reveal the QR code when pushed.

The advantage is that you can place the switches anywhere you want without having to rewire or reconfigure anything. Also, the cameras can easily read QR codes and the setup would be even simpler than building an LED matrix.

Of course this would require access to 3D printers but eventually these switches could be mass-manufactured in China

[yogi149]

So Winghunter, somehow I have a problem with understanding here.
You want switches on panels to switch LEDs, which you then identify with a camera in order to switch inputs in the simulator.
To do this, you have to control the LEDs using switches and a regulated current. Preferably with constant current sources.
Then you have to put that on a circuit board, when you can simply connect or solder a switch. It's not an LED matrix.
With the planned number of inputs, it will be a huge circuit board and a huge pile of cables.
A transmission with an optical status table is a nice idea, but where exactly is the simplification?
There are different bus systems such as I2C or CAN bus for more inputs. Even USB with named boards is easier to manage.
And scanning at 60Hz is unnecessarily fast for switch panels.

I myself took the Arduino mini with I2C bus 5 years ago to control instrument replicas, all with a mega as a central computer connected to the simulator via Ethernet.

[winghunter]

The idea is to connect one switch to one LED, just as you connect one switch to one Arduino input. I don't see how this would add more cables or even require a circuit board ?

The optical transmission is not to bridge distance, but no remove the limitation on the number of inputs per USB device. I.e. a single usb device can now handle 100k inputs. If you can find the muse and space to solder 100k LEDs that is.

[gwheeler]

53 minutes ago, winghunter said:

The idea of using QR codes on switches is growing on me. These could be 3D printed and reveal the QR code when pushed.

The advantage is that you can place the switches anywhere you want without having to rewire or reconfigure anything. Also, the cameras can easily read QR codes and the setup would be even simpler than building an LED matrix.

Of course this would require access to 3D printers but eventually these switches could be mass-manufactured in China

My model isn't using QR codes for individual controls, but rather as a registration for the panel itself. So QR codes in opposite corners identify the panel, and that Id maps to a profile which contains information about the panel's dimensions and controls (control type, position, state-to-action mapping) - each panel has a local coordinate system where the QR represents an origin corner and there's a global "console" representing the camera's total FOV. When you drop a panel into a slot in the console the software registers which panel was placed, uses the QR codes to determine the bounding box within the camera FOV to associate with the panel, and uses known information about the panel's dimensions to figure out the necessary offset values so it can translate panel-relative component coordinates to absolute viewport-relative coordinates, re-generating a global lookup table. This would be necessary I think because if the camera is centered underneath the console, an inch in panel space will take up a different amount of pixels in the camera's FOV depending on how close to the center the panel is located. This de-skewing is of course more complicated from a software standpoint but once the functionality is established it would allow unlimited permutations of drop-in panel arrangements and the only coordinates you have to worry about when mapping controls are panel-relative. 

As far as tracking state of individual components, I think it would be down to simple visual indicators. Rotary switches could have a high-contrast arrow or something on the underside that directly shows the panel-relative angle of the knob (albeit mirrored.) Rotary encoder functionality could be achieved by comparing adjacent frames for state changes to determine direction of rotation. Buttons could be designed with some sort of push-rod mechanical apparatus that reveals or changes a visual flag when pressed/latched (I could see a bolt rotating as its cams engage with spiral grooves in an outer housing as one design that would be compact in the horizontal plane) - ideally production versions of controls would derive from a library of just a few universal 3D-printed mechanisms, with interchangeable caps/knobs/etc.

Obviously none of this is practical for a cockpit that's only meant to closely replicate one specific platform, but the up-front complexity of the image processing solution could yield a system that allows incredible versatility and the ability to change between fairly high-fidelity cockpit layouts within minutes.

[winghunter]

Before I add more confusion to the LED solution, here is a simple sketch of how I intended the wiring. As simple as possible, so everyone can wire this without needing a printed circuit board. And here is a link to a live circuit https://tinyurl.com/y2opwf9f

Edited December 14, 2021 by winghunter

[winghunter]

55 minutes ago, gwheeler said:

My model isn't using QR codes for individual controls, but rather as a registration for the panel itself. So QR codes in opposite corners identify the panel, and that Id maps to a profile which contains information about the panel's dimensions and controls (control type, position, state-to-action mapping) - each panel has a local coordinate system where the QR represents an origin corner and there's a global "console" representing the camera's total FOV. When you drop a panel into a slot in the console the software registers which panel was placed, uses the QR codes to determine the bounding box within the camera FOV to associate with the panel, and uses known information about the panel's dimensions to figure out the necessary offset values so it can translate panel-relative component coordinates to absolute viewport-relative coordinates, re-generating a global lookup table. This would be necessary I think because if the camera is centered underneath the console, an inch in panel space will take up a different amount of pixels in the camera's FOV depending on how close to the center the panel is located. This de-skewing is of course more complicated from a software standpoint but once the functionality is established it would allow unlimited permutations of drop-in panel arrangements and the only coordinates you have to worry about when mapping controls are panel-relative. 

As far as tracking state of individual components, I think it would be down to simple visual indicators. Rotary switches could have a high-contrast arrow or something on the underside that directly shows the panel-relative angle of the knob (albeit mirrored.) Rotary encoder functionality could be achieved by comparing adjacent frames for state changes to determine direction of rotation. Buttons could be designed with some sort of push-rod mechanical apparatus that reveals or changes a visual flag when pressed/latched (I could see a bolt rotating as its cams engage with spiral grooves in an outer housing as one design that would be compact in the horizontal plane) - ideally production versions of controls would derive from a library of just a few universal 3D-printed mechanisms, with interchangeable caps/knobs/etc.

Obviously none of this is practical for a cockpit that's only meant to closely replicate one specific platform, but the up-front complexity of the image processing solution could yield a system that allows incredible versatility and the ability to change between fairly high-fidelity cockpit layouts within minutes.

I really like this, because its a completely wireless and very user-friendly solution. And USB cameras for this usecase are even less expensive, around $5. Additionally people can completely create their own switches while with electronics this is much harder to do.

 

I mean people could even make switches and knobs out of wood. Also great for kids to hook up their toys to the computer.

 

Edited December 14, 2021 by winghunter

[gwheeler]

31 minutes ago, winghunter said:

I really like this, because its a completely wireless and very user-friendly solution. And USB cameras for this usecase are even less expensive, around $5. Additionally people can completely create their own switches while with electronics this is much harder to do.

 

I mean people could even make switches and knobs out of wood. Also great for kids to hook up their toys to the computer.

 

 

Where are these $5 cameras of which you speak? They'd have to be high-res and wide-angle to capture a console's worth of controls without getting into the complexity of an array with overlapping visual fields.

[yogi149]

Hi Winghunter

vor 52 Minuten schrieb winghunter:

a simple sketch

yes, I thought, that you want use only resistor for the LED current. But if you want more than 100 hours of livecycle for the LED panel, you should use LED-sources with constant current.

In your circuit the LED current changes with every activated LED. And normally LED driven in that manner last only 50-100 hours.

I know that because we used some RC-light modules for static models, which drove customers angry when LED died after some hours.

Best for LED are pulsed power sources. (Take some videos from modern cars LED-backlights, you will see them flicker)

It may be wireless on the PC side, but the wiring of the LED image and the image interpreter will not be easy, and certainly no less work than individual USB boards.

And why do you talk about Arduino boards, when there are BBI boards, that handle switch as input and work as joystick for the simulator.

 

My experience with cockpit makers: they want something that is plug and play. And as easy to use as possible.
And let's be honest, who wants to wire tens of thousands of switches and assign them in the simulator.

For playing with an image interpreter it is certainly a nice idea, but I don't see a replacement for the existing solutions.

And I wish you that you will reach your goal with it.

[winghunter]

54 minutes ago, yogi149 said:

Hi Winghunter

In your circuit the LED current changes with every activated LED. And normally LED driven in that manner last only 50-100 hours.

Ok I didnt know that. So I agree this would make the whole setup too complicated. Onecould buy commercial LED panels but they would drive up the cost.

I just need more than 32 inputs per board really, otherwise I have too many USB boards to deal with. So let me know when you release a 64 or 128 model. I would buy your 32 model for now just to try it out.

Edited December 14, 2021 by winghunter

[winghunter]

1 hour ago, gwheeler said:

Where are these $5 cameras of which you speak? They'd have to be high-res and wide-angle to capture a console's worth of controls without getting into the complexity of an array with overlapping visual fields.

I have used these cheap chinese USB cams for such projects before. The should be more than good enough for such a use case.

Its not that this would need super high res, probably even 720p is enough. Considering you cant get more than 40cm distance behind a panel. That means the switches etc will be rather large in the image.

https://german.alibaba.com/product-detail/conference-camera-for-pc-price-desktop-pc-web-cam-1080-full-hd-webcams-camaras-web-1080p-online-video-call-website-usb-camera-1600153547249.html?spm=a2700.galleryofferlist.normal_offer.d_title.547b21c8vo3lSZ

 

https://german.alibaba.com/product-detail/professional-oem-usb-2-0-web-camera-1-7mm-fisheye-lens-mini-wide-angle-1080p-full-hd-industry-pc-uvc-micro-usb-camera-60812785995.html?spm=a2700.galleryofferlist.normal_offer.d_title.547b21c8vo3lSZ

 

I.e. this is a 280p image, but already contains enough visual information to track many buttons. Remember the backside will be visually more simplistic.

 

Edited December 14, 2021 by winghunter

[winghunter]

3 hours ago, yogi149 said:

Best for LED are pulsed power sources. (Take some videos from modern cars LED-backlights, you will see them flicker)

 

What about these LEDs ? They already have built in resistors and take 5V directly from power source. They are a bit more expensive but they already come in a matrix form ready to solder.

https://de.aliexpress.com/item/1849928878.html
https://www.amazon.de/CHINLY-stücke-WS2812B-Adressierbare-Kühlkörper/dp/B01N0GIHFG/ref=dp_prsubs_1?psc=1

Edited December 14, 2021 by winghunter