Project: Lua Sim SDK and TCP Sim data transport layer

[RvEYoda]

.

 

Lua Sim SDK and TCP Sim data transport layer

 

Background

Some of you who know what Leavu/Lrm/Gear are might find this background useful.

After Helios with its superb UI made new instruments for Gear/Leavu redundant a while

back, I have been thinking about new projects to try out. Helios made me realize a few

things, among others that I should not try to make an external instrument software which is

as complicated as a small operating system and requires you to program your own gauges.

Instead my focus shifted to thoughts on generalizing and simplifying the development of

tools for others, and creating a more general data transport layer for communication

sim<->instrument tool, compatible with as many future products as possible. And being

able to load different tools and instruments from different applications, but use the same

data transport layer.

 

The two main problems I had when making leavu-like software (and I think others faced as well):

 

1.

How incredibly unpractical the lua development environment is

. Often you have to restart

the game just to reload a script or alt tab out to read a log file to see debug prints! (at least

for someone used to a nice big IDE with project overview and lots of auto-generation

features :)) and proper console output. Coding with no compile time type safety and with a

startup-shutdown time of sometimes up to a minute...

 

2.

Each new game, a new data format, a new software is made, or a new interface is added

to that sim for a specific external software. Instead the community could perhaps produce

something more general together?

 

 

 

Overview

Since a couple of weeks back I have been working on a new project, which probably won't

entirely remedy the issues above, but maybe we can try to help each other out here. Before

you think this is just a preliminary discussion: No. I already got a working prototype. ;).

It's not intended to be locked to any particular language or IDE/tool, but I happen to like

Java so that's what I have used for my client software.

 

I intend to produce some tools that I will share with everyone that will let you use lockon

/dcs/xplane as your custom lua compiler and runtime environment. This is achieved in two

ways. First the idea of saving your scripts in the script folder (i.e. config/export/) needs

to be removed. Instead scripts and custom behaviour need to load and unload dynamically when

you want it to. Second, you obviously need a way to do this, so there should be a thin tcp

server running as a standard/default script (this is the only custom lua content saved to

the config/export/ folder).

 

(IDE setup example, how I got it right now)

 

Here I have uploaded the source file to lockon, WHILE lockon is running and enabled it.

What if I get a lua error in the log here? Aw the lua code throws an error on line ...

Edit the lua source in eclipse/insertYourFavoriteIdeHere and upload the new script. Don't

ever restart fc/dcs :). I can run the uploaded script instantly or save a copy as a file on the

"server" (server = sim). Of course, your uploaded script can also be things like flipping

switches or changing map etc. (Perhaps change mission behaviour if we had such an interface

in the dcs lua environment?)

 

Ok what now, I've uploaded my script, it is extracting some data from the sim, hwo do I get

it to my other machines ?

 

Ok my proposal (and implementation) is this: Use a low level message format, (binary or string,

depending on target, lua is better at strings, binary is better after data is outside the

"lua world"), to wrap high level message contents. JSON (

http://www.json.org/

) seens particularly

well suited for changing data structures, the high level message contents (we don't want to

alter our message formats or data distribution layer just because we have a new simulator or

new export functions).

 

A few sample pictures below.

 

(General concept of the data/command io, although not mentioned here, it would work just

fine also with dcs, bms and probably also variants of MS FS)

 

(Principle of sim data/command io and distribution)

 

Example of the low level binary message format I am using right now.

NOTE: THIS FORMAT IS NOT USED FOR COMMUNICATION WITH DCS/FC LUA. IN

THAT CASE THE LOW LEVEL STRING FORMAT IS USED INSTEAD.

 

Example of the low level string message format I am using right now.

I use this format between client applications and lua, since lua 5.1 doesn't

have support for bit operations.

 

 

 

 

License

This is becoming a rather large project for me, so I'm going to put some limitations here. First,

I will share the source with everyone, eventually, but it will only be free of charge for

non-commercial purposes. If anyone does want to use it for commercial purposes (I will not mention

names here, but I have been approached regarding this) then I would be willing to license it for

you, ofc provided we can agree on price. :)

 

 

 

Implementations

I've made my implementations in java, but the data formats are all independent of

programming language and hardware, so there is really no restrictions here.

 

The lua parts are pure lua (except luasockets ofc which comes with fc/dcs).

 

Maybe I should write a bus client that subscribes to autopilot status while I'm in the bathroom? ;P

 

 

 

 

Instruments

I hope that someone will pick this tool up and make something useful with it :).

 

 

Contact

You can reach me on msn, skype or email. (or phone?)

Just PM me and I'll give you my details.

 

 

.

Edited May 11, 2012 by =RvE=Yoda

[Speed]

I was thinking of something along the lines as this Yoda, just not so advanced. I was thinking of more like a general template to start Luasocket from. I really like the ability to upload scripts real-time- though this can be implemented in other ways too, I've done it myself. I used to have a "reload slmod" command so I didn't have to restart DCS. The problem was, in the end, there were too many global variables/states that needed to be reset in special ways so I dumped the capability... not that I couldn't have made it work, it was just too much of a hassle to make it work- not worth the effort. Anyway...

 

I'm trying to understand the reason for the low level message format- how is this better than Luasocket? It is faster and more universal?

 

Anyway, a major handicap I immediately see is that it appears you are only implementing this mod for the "export" Lua environment. If you make it work out of the "net" environment instead, then it can access the "mission", "net", "export", "config", and "server" (aka "main simulation") Lua environments as well. The mod becomes vastly more useful.

 

(Perhaps change mission behaviour if we had such an interface

in the dcs lua environment?)

 

Such Lua interfaces have existed for a long time already, in the "main simulation" AND "mission" Lua environments. The catch is that ED does not officially support these environments, so any mods that work in them need to be actively supported by the mod creators as these Lua environments are subject to change (that said, the rate of change is fairly slow and very manageable- from patch 1.1.0.8 to 1.1.1.1, Slmod required no changes for compatibility- DCS World 1.1.2.0 was the first patch that actually broke something in Slmod, and the modifications required to fix the features that were broken were rather slight).

Edited May 8, 2012 by Speed

[RvEYoda]

I was thinking of something along the lines as this Yoda, just not so advanced. I was thinking of more like a general template to start Luasocket from. I really like the ability to upload scripts real-time- though this can be implemented in other ways too, I've done it myself. I used to have a "reload slmod" command so I didn't have to restart DCS. The problem was, in the end, there were too many global variables/states that needed to be reset in special ways so I dumped the capability... not that I couldn't have made it work, it was just too much of a hassle to make it work- not worth the effort. Anyway...

 

I'm trying to understand the reason for the low level message format- how is this better than Luasocket? It is faster and more universal?

 

Anyway, a major handicap I immediately see is that it appears you are only implementing this mod for the "export" Lua environment. If you make it work out of the "net" environment, then it can access the "mission", "net", "export", "config", and "server" (aka "main simulation") Lua environments as well. The mod becomes vastly more useful.

 

 

 

Such Lua interfaces have existed for a long time already, in the "main simulation" AND "mission" Lua environments. The catch is that ED does not officially support these environments, so any mods that work in them need to be actively supported by the mod creators as new patches change the way these environments work.

 

No you misunderstand: the low level binary format is not a replacement for sockets. It is type of encoding of what is sent over sockets - BUT NOT TO DCS/LOCKON :).

 

I've made two low level message formats:

1 binary format

1 string format

 

I use the binary format all the time once data is outside lockon (between my different computers and programs)

and I use the string format to communicate between lockon's lua and my translator application: And this is sent over luasockets!:smilewink:

 

---------|----- STRING FORMAT ----|--------------- BINARY FORMAT ---------------------------|

sim <-> lua <-> luasockets <-> translator <-> data bus <-> data subscribers and distributors (the translator can be both)

In another example with a program which does not use lua but dll hooks:

-------------|------------------------------------------- BINARY FORMAT ------------------------------|

sim <->C/c++ dll <-> sockets/shared mem <-> translator <-> data bus <-> data subscribers and distributors

 

You can install this hook into _any_ lua environment supporting luasockets. It turns _any_ lua environment into your own personal compiler :).

You may inject _any_ script you wish ;). Obviously though, the environment needs some cycle/iteration hook to use.

 

There is also multi-client support so multiple sessions may connect to the lua tcp server simultaneously and feed it with different scripts. Each client

object on the lua side maintains its own set of scripts. (except for those saved on disk obviously ;))

 

From lowest to highest level:

sockets <- luasockets <- low level message format and protocol <- high level message contents <- data/script contents

 

 

Example input from java:

 

   @Override
   protected void onConnected(final long timeMillis) throws IOException {
       super.onConnected(timeMillis);
       try {

           // Upload and load all scripts we need
           uploadScriptFile("lua/lockon/FlightDataExtractor.lua");
           loadUploadedScriptFiles();

           // Activate cycle callbacks
           clearCycleables();
           addCyclefcn("exportFlightData");

       } catch (final SizeLimitExceededException ex) {
           ex.printStackTrace();
       }
   }

The flightDataExporter example above sends a JSON-formatted, LL string format wrapped message

 

Example msg input to java from lua

   @Override
   protected void onNewMsgs(final ArrayList<ParsedMessage> msgs, final long timeMillis) throws IOException {
       super.onNewMsgs(msgs, timeMillis);
       for (final ParsedMessage pm : msgs) {
           final String msg = new JLuaSockIfMsg(pm).getDataAsString();
           if (!msg.equals("HEARTBEAT\n") && !msg.equals(equals("EXIT\n"))) {
               try {
                   final JSONObject parsed = new JSONObject(msg);
                   final Map<String, Object> map = Utils.JsonObj2Map(parsed);
                   Utils.printMap(map);
               } catch (final JSONException e) {
                   e.printStackTrace();
               }
           }
       }
   }

 

...*which can be made into human readable form: http://gigurra.se/loXport.txt

 

 

.

Edited May 8, 2012 by =RvE=Yoda

[Moa]

Hi Yoda. Good to see you are alive and well and modding.

 

I think what Speed was trying to point out was that in DCS the different scripts (called "environments") are prevented from accessing each other's data - apart from some massively inconvenient tricks.

 

Your data bus is an excellent idea. Mucking around in Lua is a complete pain and the lack of decent messages makes it very unproductive. Something all the eager beavers porting their aircraft to DCS:World will soon learn (although no doubt many projects will succeed through sheer force of will).

 

I do have a moving map control for FC2 that would be an ideal candidate for using this bus. It uses Igormk's excellent TC-1 map. The map control works fine for FC2 but I'm trying not to mod for ED products anymore (the unproductiveness of Lua makes me grit my teeth and the recent firewalling of "environments" drives me batty), so I don't know whether it'll still work for DCS.

 

I have been intending to put the map source code up (Java and Lua, of course) for use. This might be a good candidate for testing your data bus.

 

Personally I prefer XML to JSON, since Java handles XML in the standard library (JAXB is awesome once you get your head around it) and for small messages the size is the same on the wire (since a single TCP packet could be involved in both cases, thanks to Minimum Transfer Unit sizing). The inefficiency of XML would not be apparent for small messages, although testing would be required to see whether the processing time for large messages would be significant or not (often, just by caching the XML marshaller and unmarshaller you can speed XML up enormously).

[metalnwood]

I too have been down this track a couple of times, once was with DCS when I was making some ipad controls and the other time was with FSX. Actually before the days of J2EE I had to create message busses and adaptors more times than I could count but those days are gone now..

 

I am even looking at it right now so that for my hardware panels I have a standard way of plugging in to different sims. For DCS it's the lua exports, for which there is no standard to export the data, just a method - i.e. lua sockets to spit it out. For FSX I am looking at using FSUIPC, the same for xplane and that covers my immediate needs.

 

So, my question would be, what are your goals here? Are they to try and get all developers wanting data out of DCS to have a single defined way of doing it so helios, tars, x, y and z are not all meddling and overwriting each others export scripts? I see something like that to be fairly valuable and it's the kind of thing that you would find would get a group effort to keep it up to date as the simulator changes.

 

There is a difference between the protocol, software design and the implementation. From a developers point of view are you saying that you would license the software which is fair enough as it's your implementation, or would you license the protocol as well.

 

My thoughts are that anyone can sell anything they develop, or give away etc but I think if we are going to have some sort of data protocol out of DCS then that should be a well thought out piece of work with no restriction on it's use. If there is restriction then I don't see how adoption will work very well for commercial people, that could include people asking for donations, not actually taking money for all downloads.

 

How many of us are in this boat at the moment? Not quite sure, I can think of gadroc with helios, tars, the ipad apps, myself, nothing else pops to mind.

 

To be honest, the biggest pita I have with DCS is the complete lack of proper documentation. If it was documented then getting the information out would be a snap and I personally dont find the small amounts of code to export the lua the problem. It's just finding where the data lives in the variables and making sense from it. To date the community has shared a lot there with different people digging up different things.

 

When I was doing some FSX stuff I found the simconnect fairly bad and decided to use FSUIPC. You can have some commercial arrangement with Pete on that but I found that the easiest way was to ensure that people needed to have an existing licensed version of the software. That way they could factor that in to the cost and it didn't mean commercial stuff on my side. In fact all my beta users already had fsuipc so it wasnt a problem for them.

 

Not sure where I am going with this post now, I think the concept is good but if it is to try and get all developers in the same boat then this kind of things has to be the 'free' sort of thing otherwise commercial guys are just going to do what they have always done.

 

I hope I am making this sound right, I dont mean to say people should give stuff away for free but it depends on your goals, if your goals are to band the developers together then I just dont think it will catch on far enough if there are $$ attached to it because developers can do it themselves. If the goal is not to unite the developers but release a tool to help some developers then I think that is a good thing, $$ attached or not, they can make up their own mind about it.

[metalnwood]

Just to add to that, I do think it's a shame that ED are not doing anything about this.

 

I have emailed them and I know a couple other people who have been wanting to develop have emailed them as well and either got no reply or in my case I did get a reply but what I would consider to be fairly standard, probably a cut and paste job.

 

Now it's clear that ED are engaged with third parties to develop for DCS but it looks like that effort is to develop content not necessarily the huge number of external applications, hardware etc that is available for all other manner of sims from racing to flying.

[Tex-Twil]

Hi,

this looks interesting but I wonder how easy/difficult would it be for a 3rd party instrument application to use your protocol. Couldn't we get a kind of preview version to see how the client side (e.g json) work ?

[RvEYoda]

Sorry everyone I have not answered. Since i did not see any replies I thought nobody was interested :), and I only visit these forums quite rarely nowadays.

 

Basically I made a working proof-of-concept setup, by using the transport layer described above coupled to a general lua table->JSON format converter.

 

I do not intend to put any license on the protocol nor the lua side of things. Though I might put some restrictive license on for example java implementations for the other side.

 

If you would like to try the current implementation then please contact me on msn (and I should be able to answer most times of the day)

 

ancient_banana_tmuk ------ AT _____ hotmail===dot===com

 

Don't try to email this address though, it's only for msn.

[leonpo]

Is this project is still available? Sounds very interesting

 

Leon

[trujillostm]

LUA + Java...

 

Hello. I have seen your post about using Java (with Eclipse) to read LUA commands from DCS. Can you say me how to do with FSX (Flight Simulator X)? My idea is to make I touchscreen software in Java.