Another disruptive technology...

[geneb]

As most of you know, I'm the kind of flight simulator geek that turns the knob to 11 and snaps the thing off. I'm often used as an example when arguing with a spouse - "See honey, it's not that bad! I could be doing what geneb is!"

 

Yesterday I had an email conversation with Jeri Ellsworth, the hardware engineer behind an augmented reality product called castAR.

 

What I learned took everything I thought possible with flight simulator display systems and upended the box into the street.

 

I thought the DIY collimated display system that my friend Wayne & I built was the pinnacle of home cockpit building. Boy was I wrong.

 

You've read about Flim and others building display "cubes" to use with DCS - it takes 5 or 6 projectors, a ton of room and even more money.

 

You've read about people building instrument panels with expensive, special sized LCD panels in order to display MFDs and other avionics.

 

With castAR, that ends. You don't need the projectors any more. You don't need the LCD displays for your instruments any more. All you need is a castAR and some retro-reflective screen material.

 

The castAR is augmented reality in the most literal sense. The glasses you wear sport a pair of 720p pico projectors and a head tracking system that's good to .07mm of positional accuracy. Those projectors don't put out much light - in fact, unless you're looking at a retro-reflective surface (think movie screen) you won't see what they project at all.

 

Now imagine a cockpit environment where the cockpit is surrounded by this retro-reflective material (it's a reasonably priced fabric). Every where you turn your head, you're going to see the outside view, correctly oriented and focused.

 

Inside the cockpit, instead of a thousand dollars or so of LCDs for the MFDs, HUD, etc, you've got little squares of retro-reflective material inside the bezel frames of the avionics. The castAR knows where you're looking and it can show you the avionics in the cockpit right where they should be.

 

Here's a demo video Jeri posted today:

 

After watching that, I discovered (once again) that it's impossible for a grown man to retain any dignity while giggling like a little girl.

 

Because of how the material reflects light, it's possible for two or more people to share the same reflective surface. This means that doing tandem cockpits is now a realistic choice.

 

The Oculus Rift is a cool device with some interesting VR applications. Unfortunately, it's totally useless if you have to interact with both a physical and virtual environment at the same time - you try manipulating cockpit controls you can't even see! When the castAR hits the streets, the Rift is going to be a cute, but obsolete technology.

 

I'm really hoping the guys at Eagle Dynamics will add castAR support to DCS. From what I've been told, support is a pretty straightforward thing to add.

 

Head over to http://www.technicalillusions.com and join the forum. There's going to be a Kickstarter for this come October 15th.

 

I now fully understand how fanboys are made and I've been mumbling "TAKE MY MONEY!" A LOT lately. :)

 

g.

[MACADEMIC]

Hi geneb,

 

This is very interesting! I'm not sure however I completely get the principle. I understand you're wearing shutter glasses, and there are two micro projectors seated above each glassframe projecting onto a retro reflective surface, the light reflecting straight back to your eyes, information reaching each sequentially. There's a tracking system to determine where in the scene you're looking at. But what's the camera doing?

 

The information is then projected onto a retro-reflective playing mat which then reflects back to a camera, producing a clear 3D, HD 720 image per eye with 24 bits of color.

 

?

 

MAC

[geneb]

To be honest, I have no idea. I'll ask! :)

 

[edit] Can you point to me where you found that quote? AFAIK, there's no camera mounted to the castAR itself. I think the camera you're referring to may be the one used to do the video....

 

 

g.

Edited September 27, 2013 by geneb

[hegykc]

So one could build a window frame of a certain cockpit, and cover the windows with this fabric, and be able to have the view a real pilot has? AND see the simpit controls?

 

Is this everything I've been praying for??

 

EDIT:

Looks like it might really be the answer for us flight simmers:

http://www.tested.com/tech/concepts/455605-hands-castar-augmented-reality-glasses/

Edited September 27, 2013 by hegykc

[geneb]

Yes to both questions. :)

 

Could you just imagine the fun you could have with these things? Add to Aces High or War Thunder...*wistful sigh*

 

g.

[acino]

It looks interesting, but I am somewhat concerned about peripheral vision. If you see the rim of your glasses and then "real" reality outside, it could be quite immersion breaking, I would perceive.

[MACADEMIC]

To be honest, I have no idea. I'll ask! :)

 

[edit] Can you point to me where you found that quote? AFAIK, there's no camera mounted to the castAR itself. I think the camera you're referring to may be the one used to do the video....

 

 

g.

 

2nd paragraph here

 

MAC

[geneb]

It looks interesting, but I am somewhat concerned about peripheral vision. If you see the rim of your glasses and then "real" reality outside, it could be quite immersion breaking, I would perceive.

 

Keep in mind that they're showing prototypes right now - the production part may wrap a bit more around your eyes. If it doesn't, I'll stick a bit of cardboard in there. :D

 

g.

[geneb]

2nd paragraph here

 

MAC

 

Thanks Mac, I'll ask.

 

g.

[RagnarDa]

Wow that's really clever! Talk about thinking outside the box!

[acino]

Keep in mind that they're showing prototypes right now - the production part may wrap a bit more around your eyes. If it doesn't, I'll stick a bit of cardboard in there. :D

 

g.

 

Absolutely, it is nothing that cannot be fixed by wrapping the glasses to the side and perhaps a second set of "projectors". It is just a matter of whether they will bother with that. :)

[MACADEMIC]

Thanks Mac, I'll ask.

 

g.

 

I think I found the answer in the tested.com report posted above by hegykc:

 

Between the two projectors sits a camera that recognizes infrared LEDs positioned at corners of the reflective surface. That camera triangulates the position of your head, so the software knows what angle you're looking at the surface from at all times. The games use that information to simulate fixed 3D objects in the space in front of you, so you can move your head around them and see objects from different sides.

 

MAC

[MACADEMIC]

From the geneb's first post:

 

Inside the cockpit, instead of a thousand dollars or so of LCDs for the MFDs, HUD, etc, you've got little squares of retro-reflective material inside the bezel frames of the avionics. The castAR knows where you're looking and it can show you the avionics in the cockpit right where they should be.

 

That's right, but not to forget you won't be able to reach out to turn knobs, flick switches etc. in a setup like this, without your hands getting in the way of the projectors ;)

 

MAC

[cichlidfan]

The camera that reads the LED's is going to need a very wide FOV if you intend on having anything large.

 

 

EDIT: I will be looking at the KS with interest though.

[MACADEMIC]

Now imagine a cockpit environment where the cockpit is surrounded by this retro-reflective material (it's a reasonably priced fabric). Every where you turn your head, you're going to see the outside view, correctly oriented and focused.

 

Here I'd caution about the focal range of the projectors. Quite a range there to the various canopy surfaces, especially with turning the head and panning sideways. I don't think it could provide a focused image in that setup :(

 

MAC

[geneb]

and LOOKING outside the box. :D

 

MAC - the camera they're referring to is the black disc that sits between the two projectors. It's works with the motion tracking, but I waiting for more detail on it.

 

g.

[geneb]

From the geneb's first post:

 

 

 

That's right, but not to forget you won't be able to reach out to turn knobs, flick switches etc. in a setup like this, without your hands getting in the way of the projectors ;)

 

MAC

You misunderstand - I'm not talking about projecting the whole cockpit from the game, JUST the displays. You'll have a physical MFD (like the Thrustmaster MFD button collars) with a card coated with the screen material on it. Since the castAR knows exactly where your head is, it can display the MFD symbology on that screen when you move your head to a position where it would be visible. Note that even if you did display all the cockpit artwork, the only thing you'd actually see are things aligned with where the retro-reflective screen material is positioned.

 

The head tracker appears to work something like a backwards TrackIR. Instead of the camera sitting on the monitor and the IR targets sit on a hat or whatever, the camera is on the castAR and it uses tracking targets that sit on the table. I found out today that multiple tracking target boards can be used in order to give you complete coverage, no matter where you put your head.

 

This means that if the bad guy goes screaming past you, you can crank your head around to keep him in sight without losing head tracking.

 

g.

[geneb]

Here I'd caution about the focal range of the projectors. Quite a range there to the various canopy surfaces, especially with turning the head and panning sideways. I don't think it could provide a focused image in that setup :(

 

MAC

 

It may not be an issue MAC. Check out this post:

 

http://technicalillusions.com/forums/thread-70-post-325.html#pid325

 

g.