VRK - a virtual reality enabled kneeboard with touch and ink support

[AMVI_Rider]

What is VRK?

Given the latest improvements of VR hardware, the most limiting factor for VR flight simulation in VR is the inability to access the documents and taking notes.

So, the name says it all: VRK or VR Kneeboard is a kneeboard application for flight simulators. It designed to be used in the VR environment where accessing paper and taking notes is annoying to impossible.

 

How does it work?

The application generates an overlay in the VR world that can be placed and sized at will. When focused (looked at) the kneeboard can change its transparency and size to increase readability.

The kneeboard is divided in tabs that can be created at will. Each tab can contain either:

• An imported PDF (useful for charts, frequency tables, briefings, etc.).
  
• An empty Notes tab, single or multiple pages per tab is supported.

PDF and Notes can be used with a pen (digitizer or pen tablet) to take notes either on existing documents or on the bank sheet.

Scrolling and zooming are controlled by gestures or physical tablet buttons.

 

Starting from 1.3 client/server mode has been remove from the application. Users that need to use client/server should stay with 1.2 that will be supported with the current 1.2 feature set.

 

Supported tablets are:

Wacom (all Intuos line is expected to work in WinTab mode)

Huion (using the custom driver, see Appendix A of the Quick start guide provided with the installer)

 

Reported working models:

Wacom: Intuos S, Intuos M (both BT and USB version)

Huion: HS64, H640P, H950P

 

XP-Pen devices are not working with VRK.

 

What else?

Here are some plans for this tool for the future:

• Use the tool during my squadron activities :pilotfly:
  
• Fix bugs!
  
• Kneeboard graphics (borders)
  
• OpenGL support for OculusVR (experimental)
  

 

Supported platforms

The tool is compatible with both OpenVR(a.k.a. SteamVR) and OculusVR. Beacuse OpenVR is the only platform supporting external program overlays "out of the box", OculusVR requires some configuration black magic.

VRK tries to simplify the process to the maximum extent, but some user patience is required to survive the initial impact.

 

Most non Oculus HMDs are expected to work in SteamVR mode. Pimax 5k, HTC Vive, Valve Index have been tested by users. All the other models of their families are expected to work as well.

 

Where to download the tool?

Like DCSMP and the other tools I designed for flight simulation, the installer is hosted by my squadron website:

 

VRK Installer Version 1.4.3.07

Discord channel

GitHub for Changelog, Issues and alternate download

 

Any comment and suggestion is appreciated, just consider that I write code and support the tool in my very limited spare time.

 

Example videos:

 

 

For those who wants to stick with 1.2 and the Windows Tablet here is the link to latest 1.2:

https://drive.google.com/file/d/1wy0M4-Vq4YC7HMq2lRNTif2YMWuUwQIv

Mind that this version is not supported anymore and the experience with tablets is not even close to the latest VRK and a pen tablet: more ergonomic, more features and much better pen-to-display latency.

Edited September 22, 2020 by AMVI_Rider

[OttoPus]

Outstanding idea, keep up the good work!

[Habu_69]

Potentially a great tool for us VR pilots. Easy to install. But how about some more detailed instructions on client/server connection. I assume one installs the program on both devices and chooses the relevant client or server mode. Then what? Wired connection or wireless? What net address? Port? What do all those SETTINGS parameters do?

[EntropySG]

outstanding idea, will try it as soon as I get home.

the link you provided seems to be broken for me, this one it is I assume

 

https://www.amvi.it/download429-vr-kneeboard.html

[AMVI_Rider]

Potentially a great tool for us VR pilots. Easy to install. But how about some more detailed instructions on client/server connection. I assume one installs the program on both devices and chooses the relevant client or server mode. Then what? Wired connection or wireless? What net address? Port? What do all those SETTINGS parameters do?

 

The client/server mode uses you home network (LAN). It is supposed that both are connected to your "home" network via cable and/or WiFi.

 

I will be back with some instructions very soon, maybe with a video if I manage to get the overlay recorded too. Frankly I'm thinking of completely changing the network setup part with some kind of automatic discovery of the two machines (for many good reasons).

 

For the connection, the only parameter you have to change is the network address field on the CLIENT machine.

In this field you have to put the LAN IP address (a.k.a internal network address) of the SERVER machine. If you don't know how to obtain it follow these instructions:

 

https://www.digitalcitizen.life/find-ip-address-windows

 

These address are usually in a form like 192.168.x.y (usually x = 0 and y is an "arbitrary" number from 1 to 253).

 

Once you put this address in the "Network Address" field of the CLIENT configuration, click OK.

Don't change the port unless you absolutely know what you are doing. In any case, put the same number in both CLIENT and SERVER.

Everything else is fine for your first test.

 

 

Once the configuration window closes, the CLIENT tries to establish the connection with the SERVER.

 

As you will see, when the connection goes up the big red cross in the server application (which indicates that the connection is down) will go away. Also, the CLIENT title bar will change showing the "CONNECTED" indication.

Re-connection in case of connection loss is automatic.

 

 

Meaning of the other parameters

Virtual Reality checkbox: enables the VR rendering on the helmet (if availabe). You have to set it on on the SERVER. Setting it off on the CLIENT is not required unless you have a visor installed on the client too.

Yaw, Pith, Roll, X, Y, Z is the pose (position+orientation) of the kneeboard in the virtual world. The zero is your head. Default values is approximately on top of your right leg.

Default/Focused Width is the width of the kneeboard in the VR. Default is when you are not looking directly at it. The application will switch to the Focused value when you are directly looking at the kneeboard.

Default/Focused Alpha is the opacity of the kneeboard with 0 meaning completely transparent and 1 completely opaque. Also in this case the two values are switched automatically when you look at the kneeboard.

Focus delay It is not used at the moment. In the future it will let you change the delay between the switch from Default and Focused representation.

 

When I say "look at the kneeboard" I mean that the kneeboard is in front of your eyes.

 

 

outstanding idea, will try it as soon as I get home.

the link you provided seems to be broken for me, this one it is I assume

 

https://www.amvi.it/download429-vr-kneeboard.html

 

Thank you, yes the URL is that one.

Hope to have some feedback from you guys! The application lacks of some polishing and maturity because I'm focused more on the behavior and use cases, time will bring all the little things.

[EntropySG]

tried standalone, cant seem to interact with it. using mousewheel or clicks but it wont flip pages. installing on my surface pro and see if it works in server/client mode with touch and ink

[lucky-hendrix]

This looks amazing !!!! Can't wait to try it !

 

Sent from my VTR-L09 using Tapatalk

[lucky-hendrix]

Is the development 'open source' ? Is there anyway we can help ?

 

Sent from my VTR-L09 using Tapatalk

[AMVI_Rider]

Is the development 'open source' ? Is there anyway we can help ?

 

Sent from my VTR-L09 using Tapatalk

 

The intention is to keep a closed source for the core, but opening the protocol as soon as it will became stable (in between RC and Release). Not because I want to protect anything (no rocket science used) but just to avoid the proliferation of forks typical of the opensouce.

Don't get me wrong on this: I fully support the opensource concept in many domains. I just think that the flight sim community is "too small" to take a real advantage from opensource. Modularity is the key.

I may be wrong, and other people are following different paths (thank you Ciribob for SRS). I will open the source the day I will not be able to support it anymore.

 

Going back to helping: any experience with iPad development? Android is not a problem, but Apple ecosystem is far from my area of expertise.

Edited August 6, 2019 by AMVI_Rider

[lucky-hendrix]

Makes sense, I was more asking to see if you need help. I would be afraid to see that kind of project slowly die because of lack of contribution

 

 

Unfortunately no, experience with apple ecosystem at all :/

 

Sent from my VTR-L09 using Tapatalk

[AMVI_Rider]

tried standalone, cant seem to interact with it. using mousewheel or clicks but it wont flip pages. installing on my surface pro and see if it works in server/client mode with touch and ink

 

Have you been able to run it on the surface?

 

I just added a link for the Quick Start Guide in the first post.

[Gianky]

Hey Rider!

 

 

First of all, congratulations for what appears to be a great application for VR users.

 

 

That said, I glanced at the manual and tried the program on my PC in standalone mode: correct me if I'm wrong, but.... you can only write on a blank page with a stylus? No typing with a keyboard? And, if I'm wrong, how do I do it?

[AMVI_Rider]

Thank you, I hope so and that's why I released to the community. I felt the need of paper in the cockpit and that's one of the two reason that made me slight VR for flying, the other is readability-resolution and I see big improvements, so I thought it was the time to sole the paper thing.

 

You are right about the keyboard, no support, but for a reason: I found it impractical in every scenario I tried it. In standalone mode the keyboard is captured by DCS and giving it to VRK means loosing control on the cockpit (also switching focus in VR is a nightmare IMHO).

In server/client it is not expected that you have a keyboard attached (and easily accessible) to the client.

Moreover, unless you connect also a mouse, the stylus is the only device that can move the cursor (and show where you are about to write) without acting on the application.

So I feel that it is better to ask pilots to use a stylus tablet or buy a stylus pad rather than providing a bad solution.

 

The standalone mode, at the current state, is of little use. It will became handy in conjunction with an USB or Bluetooth digitizer like Wacom ones (the Intuos w/ touch support seems interesting) or cheaper alternatives like Huion, XP-PEN and other chinese brands.

The only concern I have is about their sizes: small devices seems too small while medium size are slightly big in my opinion, but I need to test first.

Edited August 9, 2019 by AMVI_Rider

[AMVI_Rider]

Version 1.0.1.37 released

 

Many fixes and client/server discovery implemented

[gianlu320]

I think this is very useful.

I would like to try it but i don't have any win based tablet.

Any suggestion on a cheap but good tablet for this application?

 

Thanks

[Rymy]

I would love to see Android-version of tablet implemented. There are far more Android options available.

[AMVI_Rider]

I agree about the Android version. The fact is that the only pen based I have now is the Windows and that's why I started with it.

 

In those days I almost completed the implementation of Wacom tablet for the standalone which has some pretty nice advantages over the client/server:

• Cost: the Intuos S is really cheap compared to a tablet, the size is OK to me
  
• Precision: the pen is far more accurate that the tablet one (at least the one I have)
  
• Two buttons on the stylus and 4 on the tablet

 

The drawback is that the tablet is not touch so page switching is done with the buttons, zooming and panning with the pen. That brought the need of programmable actions for the buttons and, in the meanwhile, some code optimization of the writing panel.

Also, the tab bar is now left of the main content to gain some area for the page itself.

Anyway, I'm pretty happy with the feeling so far.

 

Coming back to Android: any suggestion for a 8" class device with an acceptable stylus which doesn't cost much?

As you may imagine, this project is heavily limited in budget.

[Rymy]

I tested application and issue for me is that Kneeboard stays alpha and doesnt matter how I change setting values.

 

There is this information in first post:

 

The good news for Oculus owners (like me) is that SteamVR is complatible with Oculus devices too. It is just required that your (flight) simulator is compatible with SteamVR and can be forced to SteamVR mode.

In case of DCS this is done by adding a command line option ( --force_steam_VR )

 

Where this --force steam_VR must be added?

[Rymy]

The drawback is that the tablet is not touch so page switching is done with the buttons, zooming and panning with the pen. That brought the need of programmable actions for the buttons and, in the meanwhile, some code optimization of the writing panel.

Hmm isn't that possible to have android application to change page by swiping screen? Like example how kindle/amazon books are done?

[AMVI_Rider]

I tested application and issue for me is that Kneeboard stays alpha and doesnt matter how I change setting values.

 

There is this information in first post:

 

The good news for Oculus owners (like me) is that SteamVR is complatible with Oculus devices too. It is just required that your (flight) simulator is compatible with SteamVR and can be forced to SteamVR mode.

In case of DCS this is done by adding a command line option ( --force_steam_VR )

 

Where this --force steam_VR must be added?

 

I created a new DCS link thet points to DCS.exe and then addded --force_enable_VR and --force_steam_VR to the target. The first forces DCS to start with VR enabled regardless the options and the second forces SteamVR mode.

 

 

In this way I have two links: one for start DCS in regular monitor mode and the second to start it in VR withouth the need to switch options manually.

 

On the alpha issue, does the size changes when you look at the kneeboard?

 

Hmm isn't that possible to have android application to change page by swiping screen? Like example how kindle/amazon books are done?

 

Well for the sake of simplicity I'm considering only three scenarios:

1. client/server with windows tablet
   
2. standalone with a pen tablet (which is only a digitizer, not a "tablet")
   
3. client/server with an android table

 

The 1 and 3 full support touch and stylus as far as the stylus is a "proper" stylus and not a capacitive pen. Otherwise the option 2 is the best one. Naviganting the documents with buttons and gestures is really easy.

 

Proper styluses, which I'm aware of, are Microsoft Surface Pen(Win), Samsuns SPen(Android), Huawei M-Pen(Android) and all of them require dedicated hardware on the tablet side.

I don't know if there are active capacitive pens with the capability to work like a digitizer and letting the application to discriminate between pen and touch input. In that case let me know and I will have a look on how to support them.

[Rymy]

If I add those command line options it still does not change 'kneeboard' view. It stays still alpha. I assume when you look at it, it should be zoomed bigger (what you have set as parameters in settings page) but nothing changes. I run it as standalone version.

[AMVI_Rider]

Well, the options are not supposed to help for the alpha problem. They are there just to start the application.

Can you post a screen shot of the settings and the VR?

Are you changing VR settings in the server side of the application?

[Rymy]

Here is attached screenshot of VRK settings. I use SteamVR to change VR settings.

 

When you are using that application do you have anything set to OVRSteam application?

 

Like I said VRK is showing what I suppose nonfocused kneeboard which is partly alpha/transparent. I assume that when I look at it, it should activate and also change size. But if I look at it, keep looking at it or try to point it with AIM it does not cause any interaction.

Edited August 17, 2019 by Rymy

[AMVI_Rider]

I never touched any OVRSteam settings.

 

What kind of Oculus do you own?

 

The "Focus" logic is mostly provided by OpenVR APIs. One thing I would like to check is the VRK log file that should report Focused state if the conditions are detected.

 

It is located in C:\Users\<username>\VRK\Log.

You may delete everything, start VRK, look at the kneeboard then close everything and send the file so I can check if it is the detection or the parameter switch that is not taking place.

 

In the upcoming version (which will be 1.1) I added some debug information to try debug your situation.

[Rymy]

I did as you requested and this is only thing I get.

 

2019-08-18 15:08:57,142 [1] INFO VRKneeboard.App - Application Startup

 

I did noticed that there was one focused event on log from yesterday as I was testing different things and tried to get VRK to function. I have tried changing DCS client to full screen and window (even though in VR mode it does not seem to make any difference)

 

My headset I HTC Vive Pro, not Oculus.