Augmented Reality in Flight Simulation

[Mike Powell]

The topic of virtual reality and augmented reality in flight sim pops up frequently, generally with discussions about its feasibility, and how physical cockpit components might interoperate with head mounted displays. L-3 Link delivered such a system to the US military for training about a dozen years ago.

 

I'd vaguely remembered reading about it, but couldn't find where I'd read it. I finally ran across it again, and given the VR/AR interest here, thought I would post about it.

 

The system is called Aviation Combined Arms Tactical Trainer (AVCATT). Here's a link to a video. https://www.youtube.com/watch?v=f1jgGuOi49U The cockpit description starts at about the 5:30 point.

[Warhog]

Well that was a rather interesting video... COMBINED ARMS! eh! It opened my eyes to even more possibilities.:pilotfly:

 

OMG "the door gunner simulator" :gun_smilie: I definitely have to have one of those. " you just don't lead'm as far" :megalol:

 

And to boot, I finally found a use for this little puppy. I built it a few years ago and, after turning some heads at the range, it just sits in my play room collecting dust ("yes there's a gun lock on it")... so maybe I could use this beauty instead of building an M-60.:thumbup:

 

 

 

Yah, I know its a little dated (1917 to be exact) but, what the hell...it still a machine gun. :megalol: OK...stop with the comments... I'll build an M-60. I just need to get some detailed drawings and I'll be "good to go". Give me couple of weeks or by the time Oculs Rift has a consumers version of their product :smilewink: and I'll have one of these (see link below :yes: ) in the basement for my simulator and I can save my 1917 machine gun for ARMA... (WWI):lol:

 

https://upload.wikimedia.org/wikipedia/commons/thumb/6/65/US_Navy_030309-N-4142G-005_Aviation_Warfare_Systems_Operator_Jon_Leney_checks_a_M-60_machine_gun_during_a_pre-flight_inspection_on_an_SH-60_Seahawk_helicopter.jpg/1280px-thumbnail.jpg

 

I also have to show this video to my wife. She wasn't very happy to learn I was, in fact, building three cockpits... but they'll be just like the ones in the video... Easily converted from one style of aircraft to another and another. And they said it couldn't be done.

 

hmmm! I may need to increase my cockpit budget after this.:music_whistling:

[Deadman]

It seam the key to this system is accurate replica switch placement on panels. Combined with the see through helmet display (6:14) to be able to transition from cockpit display to the outside view. ( an augmented reality system not a virtual reality system) This sound like a very good but expensive system for it to have the necessary resolution. But hey that is just my opinion.

[Mr_Burns]

Yeah, notice how the panels still have to look like real backlit panels in the right placement, so VR might be your thing, but hotas will be the only tactile HMI possible.

 

You would even know what OSB to press or UFC or CDI.

 

I didn't reall understand what he meant by see through vr headset, but assume it means once the tracker knows your eyes are looking at your cockpit it gradually becomes see through, long way off if ever for flight sim nice.

 

Not needed for race cars as F1 is all hands on wheel ang gears or shooters.

 

That video looked like what DCS wants to be!

[Cripple]

Yeah, notice how the panels still have to look like real backlit panels in the right placement...

 

Correct... but with not to the fractions of a millimetre with the labelling in the correct Mil-Tech 101 (or whatever) font. Which seems to be what a lot of the nit-picking in some sections of the community is about.

 

At the risk of offending any partisan defenders of each particular faction: VR, as it stands now, is like trying to type a letter in a diving mask and neck brace - great for seeing *around*, you but not so great for seeing where your fingers are. Tri-monitors are like playing the piano in the bay window of your living room - you can see your fingers perfect and a view out the front, but if you look far enough over your shoulder then you still see the sofa and the cat.

 

Both of these are (like many other aspects) *compromises* forced upon pit builders by technology/funds. Which one you are in favour on is largely a matter of what you are willing to do without... If you are a warbird (or helo?) pilot with a relatively limited number of controls to guddle for, then the idea of being able to see in a full 360 with 6DOF is like Christmas and Early Retirement in one; even if it means not being able to see your (real) hands. On the other hand, if you enjoy modern fast jets with their myriad of clicky-buttons and tactical options, then you are going to consider doing that with your (natural) vision removed to be madness; and will accept a reduced field of external view in exchange.

 

(The resolution "issues", which I am *not* going in to, can also largely be seen as a matter of preference relating to one's chosen milieu. The further away one's opponent can launch a deadly attack upon one, the more likely one is more concerned about resolution up front (or on the radar screen) rather than 1:1 360-degree vision through the bubble.)

 

VR is not AR... yet. AR is kinda what VR will grow up to be when it's mature. I don't think anyone is claiming that the Rift et al are the final expression of the technology , far from it. However they seem to be sufficiently interesting to be worth a punt. AR, when it becomes available to mere mortals, should be the best of both worlds - the fidelity of a 1:1 functional cockpit *plus* the 1:1 360-degree 6-DOF view of the outside (virtual) world.

 

Until then, pick the immersive factor *you* need the most - which doesn't necessarily mean the differing perspective is wrong...

 

 

 

Oh, and as a final aside I was washing my face this morning. With a wet face and my eyes tightly shut I: turned off the correct tap (on my first attempt), turned 90-degrees (in the right direction), walked three paces to the towel rail (without tripping over the toilet), grabbed the hand towel (without any messing about), and dried my (own) face before opening my eyes. Not tricky.

 

My point is that one readily completes tasks without visually confirming the location of one's hands and fingers. I fail to see how the manipulation of (familiar) cockpit controls should be any more complex. Surely the reason *why* even 40s era aircraft have different shaped knobs (oo-er!) on the various levers is so the pilot is not required to visually confirm which knob he is fondling in the dark, and can rely upon his other senses - allow for a more efficient "eyes-up" scan pattern.

[Hansolo]

OMG "the door gunner simulator" :gun_smilie: I definitely have to have one of those. " you just don't lead'm as far" :megalol:

:megalol:

Now that very nice movie reference :thumbup:

 

Thanks a lot for sharing Mike.

 

I would be interested to see the AR from the inside. I didn't really understand how they technically do the see through thing but it did look there was at least an AN/ARC-164 digital version in the pit. Might be just as Mr. Burns suggested that the imagery is cut off when looking down. On this datasheet (https://www.link.com/media/datasheets/Link_AHMD_2011.pdf) page 2 it looks like the projection in the F-16 simulator is on glass. Does look really neat but I guess a bit out of my price range :cry:

 

For now I think I will just keep to my pit build. I may be able to write stuff without seeing what I write. But I struggle a bit with read the text of a CAS brief without seeing the letters.

 

It's going to be interested to see what the future brings for home cockpit builders :thumbup:

 

Cheers

Hans

[Deadman]

HMA he seams to just have issues with replica builders that want an out side the cockpit view. Lets let him finish his build and rebuild it once be fore we pay to much attention to him :music_whistling:

[Cripple]

HMA he seams to just have issues with replica builders that want an out side the cockpit view. Lets let him finish his build and rebuild it once be fore we pay to much attention to him :music_whistling:

 

To *whom* might you be referring, exactly?

 

 

 

Regardless, that could come across as a rather elitist and unhelpful comment. It seems that you are insinuating that someone's opinion is not valid unless he has constructed *two* simpits. Which, whilst not likely to encourage anyone perceived to be a "newbie" to venture an opinion (valid or otherwise) in this forum, is also logically inconsistent. The statement, regardless of your intention behind it, reads like "a man's not a Real Man until he's on his *second* marriage!" Personally, I'd rather get it right the first time...

 

 

I am pretty sure that's not what you meant though, and you will clarify this in due course. :music_whistling:

[Mr_Burns]

there are a huge mix of skills and desires on this forum, from simple boxes of switches to augment some keyboard presses to ful on mil spec simpits and every scale in between.

 

I think, and correct me if I am wrong, but a previous thread was hijacked by someone essentially say you are wasting your time building sim pits because of VR.

 

What this thread shows is even the military cannot replace training Emersion with VR, you still need to simulate the actual pit and the VR need to stop where you are looking at physical hardware be it tank or aircraft.

 

However, whatever your level of emersion VR is OK for some and pit building for others, so in my opinion, unless English wasn't his/her language, I took the comment why are you building panels as VR is here as a bit inflammatory similar to bagging someone who wants to make an accurate pit for their level similar to bagging someone who makes an ACHP out of a shoe box.

 

And this thread shows VR is not a place in cockpit builder thread anyway until we get our hands on these military beauties!

 

IMO

[Deadman]

To *whom*?

Well obviously i was referring to you with out any reference to your joining date or number of posts. The post was meant to give you some time before judging you one way or the other.

 

You seam to be inconsistent in your position on what you wish to build You begin with a 1:1 replica with the use of a HMD sounds great good luck I hope it works out for you.

In another thread you seam to throw that idea out the window and say "simulation, not replication; immersion, not perfection." Hum sounds like your all of a suddenly not concerned with any accuracy in what you will put together that is fine as well.

 

I have all ways been a proponent of decide what type and fidelity cockpit you can afford to build and go for it. I have cautioned people the most expensive part of building is the rebuild though so try and start with a 1:1 scale replica.

 

Your statements "Correct... but with not to the fractions of a millimetre with the labeling in the correct Mil-Tech 101 (or whatever) font."

 

"Which seems to be what a lot of the nit-picking in some sections of the community is about."

 

Came over quite negative and a bit arrogant to say that those of us that wish to build replicas tend to Nit-pick (to be excessively concerned with or critical of inconsequential details. )

 

It does seam that you have little to no experience with cockpit building. So how would you know what is an inconsequential details and what isn't?

 

You don't have the back ground to make this decision you admit that your self in your first post.

 

"talk is cheap, so here are a few pics of the build so far. So far, so good, particularly as I'm a nurse ("Jim") not a joiner... "

 

So Jim you might want to take a step back stop being quite so aggressive and just share what you want to do and how you do it if not well that is fine as well. I would make one suggestion the asterisks look a bit off the wall on each side of some words.

asterisk

 

Read more: What does the asterisk sign mean in texting - What does the asterisk mean on network solutions email :: Ask Me Fast at http://www.askmefast.com/What_does_the_asterisk_sign_mean_in_texting-qna172789.html

 

asterisk

 

Read more: What does the asterisk sign mean in texting - What does the asterisk mean on network solutions email :: Ask Me Fast at http://www.askmefast.com/What_does_the_asterisk_sign_mean_in_texting-qna172789.htmon the words *Look abit straing*

 

 

To *whom* might you be referring, exactly?

 

 

 

Regardless, that could come across as a rather elitist and unhelpful comment. It seems that you are insinuating that someone's opinion is not valid unless he has constructed *two* simpits. Which, whilst not likely to encourage anyone perceived to be a "newbie" to venture an opinion (valid or otherwise) in this forum, is also logically inconsistent. The statement, regardless of your intention behind it, reads like "a man's not a Real Man until he's on his *second* marriage!" Personally, I'd rather get it right the first time...

 

 

I am pretty sure that's not what you meant though, and you will clarify this in due course. :music_whistling:

[dotChuckles]

Asterisk around a word denotes emphasis or a bold font where a text editor cannot provide one. I forget which text editor used it as a font switch. It has become fairly common shorthand now for emphasis and tone of voice for online communication.

[dotChuckles]

But to bring this back on topic. Have a look at Hololens. https://www.microsoft.com/microsoft-hololens/en-us

 

I saw a live demo of this and was utterly blown away by it. It's bordering on witchcraft. This *may* be the answer to combining a wrap around view with a physical pit. ;-)

[Mike Powell]

Once you have adequate processing power and the ability to accurately determine the viewer's physical perspective you can do all sorts of really, really neat things. Several years ago there were academic projects that used chroma-keying and trigger symbols in the visual environment coupled with head mounted cameras and displays. The cameras provided the computing hardware with an "awareness" of the viewer's field of view and the trigger symbols and/or chroma-key indicated what and where to paint chunks of virtual reality imagery. For example, you might have a sim cockpit with green board for windows. If the "window had, say, an "@" on it, the VR image was the front exterior view. A "1" might indicate left side view. IIRC, the position of the viewer was calculated by using the images from the head mounted cameras and an knowledge of how the room was laid out.

 

What we're seeing today is a continuation of that research effort enabled by the confluence of low cost, high accuracy head tracking, cheap processing power, and cheap display systems. There are several companies pursuing their versions of how to make the next great thing. I'm pretty certain that we'll see a number of really neat & shiny things. I'm far less certain any will actually be more than a modest market success, but I do expect it to be fun watching the show.

[dotChuckles]

Mike, you may find this article interesting http://www.wired.com/2015/01/microsoft-hands-on/

 

As well as this video. While it is just a demo of minecraft. You can see where this could end up with a little vision, everything has to start somewhere.

[Mike Powell]

Thanks, yes I've been following that. I think the Tested guys also have something about it on their Youtube channel.

 

A few years back one of the Microsoft researchers posted a few short videos demonstrating some of the core functionality of the approach. I can't recall which HMD he was using, but I think the viewpoint tracking was aided with a MS Kinect. He had a virtual helicopter hovering over his coffee table. As he walked around the table the imagery he saw was adjusted to his shifting viewpoint providing a 3D walk around.

[dotChuckles]

It's exciting times certainly :-) The next decade could be a fun one!

[Cripple]

Oh dear! My apologies, Mike... I actually tried in my last post to pour oil upon the troubled waters, not petrol. :(

 

I hope you will permit me to elaborate upon a couple of points. It seems that the tone and intention of a couple of my posts has been taken the wrong way. I forget at times that this is a multi-lingual forum, and this may have contributed to the misunderstanding. Usually I am more than happy to elaborate upon and debate any contentious points in public, however in this instance I have chosen to pursue this matter in private. I will say that I find debate healthy, and any attitude of it's-my-way-or-the-highway objectionable.

 

That said, I had hoped that my somewhat tongue-in-cheek post on this thread was equally balanced between the two current stereotypes. I fear I should have made the core messages clearer, and perhaps put in earlier (and in bold type): *any* simulator is an exercise in making compromises, and the future tech will likely be a fusion of both (squabbling?) styles.

 

I have no wish to further highjack this thread, but should also clarify my comments about things being realistic enough. One of my other hobbies, which I have indulged in since my teens, is that of painting little men. You know, the inch-high metal and plastic ones. Anyway, over the years, it has become clear that "realism" is less important than what we shall call Verisimilitude - it has to *look* realistic, rather than *be* realistic. To cut a long story short, there is a tendency amongst all miniature painters (and, it would seem, simpit builders) to over-detail their work and not quite know when to stop. What I advocate in that community, and was trying to advocate in this one, is the idea of aiming for it being "good enough" rather than "perfect". Now, with a miniature I would use the Arm's Length Rule - if you hold it at arm's length and close one eye, if it looks fine then it is fine. Now, we cannot do that with simpits... but I find the idea of debating fractions of millimetres to be frankly absurd. I am aware my hands aren't what they used to be, but when working with hand and power tools I find that accuracy of +/- 1mm to be about as good as I can get it. I'd also be interested to know if people could, at normal viewing distance (which is also about arm's length), distinguish between panels that were fractions of millimetres out. The fact that the plans for these may be drafted from photos or be Best Guess makes me even more sceptical of the need for such absolute accuracy.

 

Similarly, due to usually working at a small scale, I used 1:1 simply as a short-hand for adult-sized, rather than implying I'd be getting the callipers out...

 

Thank you.

 

To get back On Topic, as the above gentleman said, the next decade looks very exciting! I haven't been this enamoured by new tech since the first VR "boom" of the 90s (which was unfortunately a bit of a damp squib). Chroma-key with view tracking. Thank you. I couldn't remember the exact terminology, so didn't mention it earlier, but My Mate Baz was telling me something about this. He's waaaaaaay more clued up on cutting edge tech than me (he works in the field, I don't), but it seems that the idea seems to be essentially "Magic Paint" that one can paint on the canopy ... and instruments! Now that was the clincher. See-through goggles, that "draw" computer-generated images over the Magic Paint - you get the full clicky-button cockpit *and* the full view.

 

( And you don't have to mess about with interfacing and servos either... Sorry, Mike. :P)

[Mr_Burns]

But to bring this back on topic. Have a look at Hololens. https://www.microsoft.com/microsoft-hololens/en-us

 

I saw a live demo of this and was utterly blown away by it. It's bordering on witchcraft. This *may* be the answer to combining a wrap around view with a physical pit. ;-)

 

Wow - I want this now - it can makes the wifes miserable face look happy for a change.

[Sierra99]

OK...stop with the comments... I'll build an M-60. I just need to get some detailed drawings and I'll be "good to go".

 

Hey Warhog...Great lookin .30! It would seem to me using something like this:

 

http://www.evike.com/products/33325/

 

Might be easier than building one from scratch. In fact, just reading the description and from what I know about AirSoft guns, it would prolly be pretty easy to tie into the trigger circuit and run the output to a Bounard board and viola.:music_whistling:

 

Sierra

[Warhog]

Hey Warhog...Great lookin .30! It would seem to me using something like this:

 

http://www.evike.com/products/33325/

 

Might be easier than building one from scratch. In fact, just reading the description and from what I know about AirSoft guns, it would prolly be pretty easy to tie into the trigger circuit and run the output to a Bounard board and viola.:music_whistling:

 

Sierra

 

Not sure if that's legal here. It may fall into the category of "replica" which is a big no-no in Canada. I will have to look into this in more detail.

 

Thanks for the heads up.