Scripting from within a mission now effectively useless in 1109

[Headspace]

I would also be very pleased when I see socket communication restored for my plugin called by Export.lua.

 

At first I did not understand what Speed meant as I could see some of my print at work in dcs.log... until I realized they were obliterated during the whole session, only to be restored at stop time.

 

But Sockets are not there either... Do other mods relying on socket work for those who are aware of this requirement for their running configuration ?

Go into more detail please. I haven't seen a cessation of sockets when I test TARS with 1.1.0.9 and it makes extensive use of Lua socket.

 

Actually, if anyone...Speed, etc...wants to throw some *technical* information my way as to what is and isn't working then go for it. I'm a little confused at some of what I'm reading about Export stuff being broken because it doesn't jive with my observations.

Edited July 25, 2011 by Headspace

[Speed]

Simple solution:

 

Those modders that have issues with existing modscripts, or specific wishes with scripts they want to implement, talk to each other about them. There appears to be a couple of you. Start a thread and discuss.

 

This will allow a couple things:

 

1) A central location where developers, testers and other interested parties can find your thought.

2) A central location for you to learn from each other about any workarounds that may exist.

3) A central location that testers can peruse towards assembling a shortlist to bring up with the developers.

 

Right now the devs have no way at all to really know what you guys use, how you use it, and why. There's no way for them to make an informed decision. This change was brought about due to the exposure of a serious security vulnerability, but there was very little in the way of resources for the developers to learn how to plug the hole yet still keep things in good shape for modders. They have a job to do which is mostly spent hunkered down over an SDK writing code. They don't have time to read scores of threads spread over many forum sections. Moderators do, but we may not have the time to really think about how each thread ties into another, and it's only one of us that has a really solid grasp of LUA.

 

So please, help us help you. Collect all your thoughts in one place. I'd recommend starting a relevant thread in the wish list section, discuss it well (and try to help us out through including some layman-friendly explanations of what things you suggest would do) and we might be able to make things happen for you.

 

But in the present chaos, it'll be extremely difficult.

 

Ethereal, this is exactly what has already been discussed amongst a few of the modders. We do intend to start a thread like this soon. But it's tough... there's just so much stuff that's being used, you know? I use dozens of different functions. It's hard to know how this stuff should be organized.

[hassata]

Lua communication to/from HELIOS is functional.

[Speed]

Go into more detail please. I haven't seen a cessation of sockets when I test TARS with 1.1.0.9 and it makes extensive use of Lua socket.

 

Actually, if anyone...Speed, etc...wants to throw some *technical* information my way as to what is and isn't working then go for it. I'm a little confused at some of what I'm reading about Export stuff being broken because it doesn't jive with my observations.

 

Don't ask me... I only started considering playing around in the export environment recently. I'd love to talk to you about what stuff you have found in there, but I know you have a little too much on your plate right now already, so to speak.

Edited July 25, 2011 by Speed

[EtherealN]

Speed, feel free to start a thread. It doesn't have to be extremely specific, just make a discussion thread and explain it's purpose in discussing what you guys use to mod things and what could help you. I'll moderate it and see if I can help you out as a communication channel whenever it's approrpiate.

 

I just want to help find a way to get a good communcations channel going, because I hope you understand that it's hard for the developers themselves to cover all bases. They're already often stuck with using their weekends to make things happen within schedule, so we have to find ways to help them.

[ivanwfr]

In fact, socket may not be at stake here. At that time, all I could see was that I had nothing getting through anymore. And without print to tell me what was working and what was not from Export.lua side, it was just an assumption that socket too was away.

 

Since then, I could replace the missing print function with io.write to trace what is happening and here is what I get: get_argument_value is the missing part and the socket code part is not even reached.

 

Code attached in PNP_ivanwfr_110725.zip.

The relevant code sample:

 

   local getArgVal	= GetDevice(0).[color="Red"]get_argument_value[/color]
   local getAltRad	= LoGetAltitudeAboveGroundLevel

   function ExportWarthog(t)

print("...TargetBridge.ExportWarthog(t=" .. t .. ")")

if getArgVal then
	print("...calling val = getArgVal(800):")
	val = getArgVal(800)
	print(".val = getArgVal(800)=[" .. val .. "]")
else
	print("...getArgVal is nil")
end

if getAltRad then
	print("...calling getAltRad():")
	val = getAltRad()
	print(".val = getAltRad()=[" .. val .. "]")
else
	print("...getAltRad is nil")
end

   end
   

 

The 1109 log:

 

    o Export_ivanwfr: dofile("./Config/Export/TargetBridge.lua")
    = TargetBridge.Start(): c:sendto(Start)
   ...ExportCoroutine(t=1)
   ...TargetBridge.ExportWarthog(t=1)
   ...calling val = getArgVal(800):
    * TargetBridge.Stop(): c:sendto(Stop)
   

 

 

The 1108 log I get from a stub that mimics a short sample session:

 

    o Export_ivanwfr: dofile("./Config/Export/TargetBridge.lua")
    = TargetBridge.Start(): c:sendto(Start)
   ...ExportCoroutine(t=1)
   ...TargetBridge.ExportWarthog(t=1)
   ...calling val = getArgVal(800):
   .val = getArgVal(800)=[0.8]
   .consoleLight=[-1]
   ...buf=[consoleLight=0.80]
   ...calling getAltRad():
   .val = getAltRad()=[100]
   .altRad=[-1]
   ...TargetBridge.ExportWarthog(t=  1): c:sendto(consoleLight=0.80,altitude=100)
   ...TargetBridge.ExportWarthog(t=2)
   ...calling val = getArgVal(800):
   .val = getArgVal(800)=[0.7]
   .consoleLight=[0.8]
   ...buf=[consoleLight=0.70]
   ...calling getAltRad():
   .val = getAltRad()=[100]
   .altRad=[100]

   (...)

   ...TargetBridge.ExportWarthog(t=25)
   ...calling val = getArgVal(800):
   .val = getArgVal(800)=[0]
   .consoleLight=[0]
   ...calling getAltRad():
    * TargetBridge.Stop(): c:sendto(Stop)
   

 

And the TARGET log from the other side of an UDP socket :

 

    led_consoleLight(0.8): consoleLight=[172]
    led_altitude(100.0) #250: ...IN FLIGHT

    led_consoleLight(0.6): consoleLight=[129]
    led_altitude( 80.0) #251: ...IN FLIGHT

    led_consoleLight(0.5): consoleLight=[86]
    led_altitude( 50.0) #252: ....APPROACH [1 0 0 0 0 B1]

    led_consoleLight(0.3): consoleLight=[43]
    led_altitude( 40.0) #253: .......FINAL [1 1 0 0 0 B1]

    led_consoleLight(0.1): consoleLight=[0]

    led_altitude( 30.0) #254: ........HIGH [1 1 1 0 0 B1]
    led_altitude( 20.0) #255: ......MEDIUM [1 1 1 1 0 B1]
    led_altitude( 10.0) #256: .........LOW [1 1 1 1 1 B1]
    led_altitude(  5.0) #257: .......FLARE [1 1 1 1 1 B2]
    led_altitude(  3.0) #258: ........DOWN [1 1 1 1 1 B0]
    led_altitude(  2.0) #259: ......LANDED :D
    led_altitude(  1.0) #260: .....CRASHED :(
    led_altitude(  0.0) #261: .....CRASHED :(

PNP_ivanwfr_110725.zip

Edited July 25, 2011 by ivanwfr

[Headspace]

ivan:

 

What control are you trying to get the argument value of?

[Panzertard]

Hello gents - I'm been away a bit, so I need to catching up to do.

Speed, you and I had a bit of talk reagrding the issues already.

And indeed - I'd like to see the modders getting started on a new thread so we can gather the request about features we need to get working, all in one place.

 

From what I know, ED will look into the print feature.

 

But regarding chatIOlib, we need to figure out how to manage the data you manipulate - how to do it most efficiently.

 

As you discovered, and we discussed over PM, the:

net.dostring_in("server", <your lua code>)

will solve most of the situations.

 

Now if we can analyze the data and feed parameters back into the MISSION using the SERVER modules we should be fine.

 

In which situations do you need to use Print() from the mission itself? Just curious.

 

Perhaps you in the SERVER-module, create a function where:

server.lua or similar modules (myChatIOlib.lua)

Function MyLogger(strInput)
 local result
 result = net.dostring_in("mission", strInput)
 net.log(result)
end

 

In you mission, you can then you could do something like:

MyLogger("print('abcd')")

[ivanwfr]

Thanks Headspace for your attention :thumbup:

ivan:

What control are you trying to get the argument value of?

If you are looking at my call to GetDevice(0).get_argument_value(800), it comes from there:

 

From file [b]DCS/Scripts/Aircrafts/A-10C/Cockpit/LightSystem/LightSystem.lua[/b]:

ConsoleLight =
{
   arg_number  = 800,
   input       = {0, 1},
   output      = {0, 1},
   radiance    = {{material = ... },
   {material   = ... }
}

 

This, and the Export.lua LoGetAltitudeAboveGroundLevel() -- (args - 0, results - 1 (meters)) which I could turn into visual cue with blinking leds ont TM Warthog throttle console as an alarm based on approach altitude -- BTW, I was just playing some fancy application, not to be seen as an attempt to provide a missing A-10 feature!

 

But, most importantly, all I've done so far is to explore what could be worth exporting data back to TM Warthog throttle, stick and console lights.

 

The main goal would be to get an overview of what could be done with some sim feed back to the input device, instead of the standard one way communication. And until previous patch, it was promising. All I'm looking for is a solution to get back on track from that point.

[Speed]

Hello gents - I'm been away a bit, so I need to catching up to do.

Speed, you and I had a bit of talk reagrding the issues already.

And indeed - I'd like to see the modders getting started on a new thread so we can gather the request about features we need to get working, all in one place.

 

From what I know, ED will look into the print feature.

 

But regarding chatIOlib, we need to figure out how to manage the data you manipulate - how to do it most efficiently.

 

As you discovered, and we discussed over PM, the:

net.dostring_in("server", <your lua code>)

will solve most of the situations.

 

Now if we can analyze the data and feed parameters back into the MISSION using the SERVER modules we should be fine.

 

In which situations do you need to use Print() from the mission itself? Just curious.

 

Perhaps you in the SERVER-module, create a function where:

server.lua or similar modules (myChatIOlib.lua)

Function MyLogger(strInput)
 local result
 result = net.dostring_in("mission", strInput)
 net.log(result)
end

 

In you mission, you can then you could do something like:

MyLogger("print('abcd')")

 

Hey Panzer,

 

I would like to talk to you at some point soon about the security measures I took with the new mod I'm working on and whether you think they are unbreakable. I think they are, but I want to just make sure.

 

Anyway, when I mention the server environment, it's not the one where that the net table lives in; no, that's the net environment. The server environment in fact refers to Scripts/server.lua, not Scripts/net/server.lua. Confusing, yes, but hey, I didn't name them :D The server environment is where scripts used to run, before patch 1109.

 

So the code you give will not work as there is no way to tell it to be passed to the new scripting environment. If there WAS a way, hell, I'd just send it:

 

function safewrite(filename, str)

local file = io.open(lfs.writedir().. "\\script data\\" .. filename, 'w')

if file ~= nil then

file:write(str)

file:close()

end

end

 

And a similar function called saferead.

 

In fact, I still believe ED should pass us functions like that- read and write functions that only are allowed to read and write to a very specific folder where no harm could be done.

 

As far as the need for print goes, how are you going to load a different mission, and have that mission run according to the mission designer's intent, if you don't get data on when certain scripts are supposed to run? That's what print is for. In the new mod I'm working on, specially-formatted messages are output to dcs.log via print. These messages are picked up and interpreted by the mod (from the net environment), where it first attempts to determine which action is being requested, and then, it takes the data (if any is required) sent, and as long as the data sent does not include various illegal things, it will take action using that data. The data or actions could be asked of any of the other four environments (server, config, export, mission), and in some functions, those environments might be required to reply in some way.

 

But without print, nothing is possible, because without a way for the mission designer to tell the mod which scripts he wants to run and what data he wants to use in those scripts, then a separate mod would have to be installed for every mission. That is not a practical solution. Currently, print is still available in the mission environment (and the mission environment is a secure environment where the io, os, and debug tables are nil), and if ED does in fact intend to return print to the new scripting environment, then this mod can work in this patch, because it's not particular to what environment these commands are coming from. Hell, you could edit dcs.log with Notepad and send commands.

 

Now this all sounds really slow and hard drive intensive, but there is a component of the mod that reads dcs.log and extracts only certain lines from it, and this component is currently written in VBscript- so DCS does NOT run this component. So the mod is effectively multi-threaded. I've stress tested the mod with dcs.log files that were artificially inflated to supermassive proportions and there wasn't an issue within DCS.

 

And of course, the whole mod is just a sloppy substitute for not having a way to pass data through memory. But what else can be done?

Edited July 25, 2011 by Speed

[Druid_]

In fact, I still believe ED should pass us functions like that- read and write functions that only are allowed to read and write to a very specific folder where no harm could be done.

For those of us who don't get the time to commit to large amounts of coding/testing etc this sure would get my vote if its possible to do.

[Speed]

Speed, feel free to start a thread. It doesn't have to be extremely specific, just make a discussion thread and explain it's purpose in discussing what you guys use to mod things and what could help you. I'll moderate it and see if I can help you out as a communication channel whenever it's approrpiate.

 

I just want to help find a way to get a good communcations channel going, because I hope you understand that it's hard for the developers themselves to cover all bases. They're already often stuck with using their weekends to make things happen within schedule, so we have to find ways to help them.

 

Yes I know... the devs need to be presented with organized wishes and lists of things the community is actually using. If no one has yet, I will be starting the thread, I hope, in a few days. To be a little more specific as to why not start it now, I'm trying to finish up a draft of a very important document at the moment for real life crap and the 1109 patch has been a supreme distraction...

need to be posting less on here for a few days as well. This would be my #1 free time priority if I truly had any freetime... still maybe I can piece something together for a post (it's gonna be a long post...).

[Speed]

I found a bit of time to work on it a little more... while we still need the print function in a bad way, at least the work-around I thought of to use until ED restores us the print funciton does in fact work:

Still many, many hours of bug-fixing and security lockouts to go in the new mod. The data passing code went from like 80 lines to like 1000 lines.... lots of room for error in a mess like that. It also takes about 4-5 secs after a chatmsg_groupMGRS is initially triggered for it to show on screen... it's a "four stage" function that the "daemon" has to handle twice.

[Headspace]

Speed:

 

What does your work-around do and how is it communicating between environments?

[Speed]

Speed:

 

What does your work-around do and how is it communicating between environments?

 

Well, you use print. print outputs to dcs.log. Print is available in all Lua environments except the triggered action scripting envrionment and is believed by Lua experts to be a safe function. ED most likely will restore it next patch to the triggered action scripting envrionment.

 

So the main part of the mod operates in the net environment, and accesses the mission, server, export, and config environments through memory with the net.dostring_in function. Appropriate function libraries are also established in the various environments with net.dostring_in.

 

So data access flows through memory when going from net to another environment. However, to get from another environment into net, such as from the mission environment into net, then the print function is used (the send_data() function to be exact). The send_data() function, common to all five environments (though it's not really needed in net), will take a table and output its contents as specially tagged and formatted print mesages in dcs.log.

 

However, that's not all... just because it's a text message now in dcs.log doesn't mean you can do anything with it. You can't even read dcs.log from within DCS because it's already open and locked. Besides, you wouldn't want to read dcs.log from within DCS... it could make the game noticably pause. So the other part of the mod, the "daemon", is a separate and nearly invisible windows service/daemon/whatever, written in VBscript, that runs whenever DCS is running, closes whenever DCS is closed, and is in a sort of hibernation state unless you are a host (it may work in single player too, and if not, I will make it work there). So when the daemon goes active (you are host), then it searches through the contents of dcs.log for any of the specially tagged messages and outputs them to another file. This other file, DCS can read, the daemon closes it when it's done writing any new lines to it, and because it only consists of mod messages, it can read it very quickly.

 

So finally, the information passing loop is completed as the main part of the mod in the net environment reads the special messages only from the small file the daemon creates, and if the format is correct, it will match a valid, pre-defined function call, AND, VERY IMPORTANTLY, if the strings and numbers do not have any attempts at escaping out of, then it will run a function. This is how security is maintained- any user defined strings will ALWAYS be encapsulated within long string brackets [[ ]] when any kind of dostring is run on them, and the square bracket symbol is disallowed in any user defined string, so any user defined string can never escape out of the long string brackets to do something malicious. Numbers are similarly treated.

 

So the path through the Lua environments for the chatmsg_groupMGRS function is this:

 

mission->(daemon)->net->(net.dostring_in)->server->(daemon)->net->(chat message on your monitor)

 

You need four separate functions to handle it now, one in mission, one in server, and two in net.

 

Also note that now, because access is made to five environments instead of just two, then a lot more capabilities exist now. A version of chatmsg_groupMGRS could be made that would read out the coordinates to you verbally, and, output to screen as white trigger text. So even more annoying :D

Edited July 27, 2011 by Speed

[GGTharos]

Why are you passing this through a file instead of a local network socket?

[Speed]

Why are you passing this through a file instead of a local network socket?

 

Because Luasocket is not available in most environments. Export is the only environment that I believe it to be available in. Maybe it's possible that I could figure out a way to put it in the others, but even if so, I'm pretty sure ED wouldn't appreciate that, as it would most probably violate your computer's security to have this mod installed in that case.

 

But I appreciate any suggestions that can be given, so thanks. I'm a noob at Lua still, so I need all the help and "why don't you do this instead"s that I can get. I'll probably be ordering the official Lua manual and reference manual off of Amazon however, just so I can have a hard copy in my hands. Nothing beats a book.

Edited July 27, 2011 by Speed

[GGTharos]

Maybe you should ask for a limited luasocket in this environment then - like, say, localhost only? :)

[Headspace]

Because Luasocket is not available in most environments.

Have you tried including it in your .miz zipfile and loading it out of there and seeing if it works? It might be worth a shot. I haven't tried it personally. Luasocket is a standalone library, you don't necessarily have to use the copy that ships with DCS.

 

Alternatively you could attempt to load it out of the path that it sits in, in the DCS install folder. Once again, I'm not in a position to attempt to test either of those things right now.

 

Running a seperate process outside of the game in order to scrape a logfile is not a realistic or workable solution to modding in the long term IMHO.

Edited July 27, 2011 by Headspace

[Speed]

Have you tried including it in your .miz zipfile and loading it out of there and seeing if it works? It might be worth a shot. I haven't tried it personally. Luasocket is a standalone library, you don't necessarily have to use the copy that ships with DCS.

 

Alternatively you could attempt to load it out of the path that it sits in, in the DCS install folder. Once again, I'm not in a position to attempt to test either of those things right now.

 

If you were going to go this route, I would say that the way that would have the highest chance of actually working would be to mod the game so that the Luasocket functions are passed it into Scripts/compile.lua, Scripts/net/main.lua, Scripts/autoexec.lua, and Scripts/server.lua.... that would get them into the four environments that don't currently have them. I can't imagine how including it in the mission itself could ever work. You can't load anything from the mission environment, it's secure. Just for example, dofile doesn't exist, and even if it did, it would probably need to be removed.

[Speed]

Running a seperate process outside of the game in order to scrape a logfile is not a realistic or workable solution to modding in the long term IMHO.

 

Well, it's not a pretty solution, that's for sure, but until something else is figured out, it's the only solution that I can think of. It is over complicated, and the data transfer is very slow. However, it does not slow down DCS that I can detect, as DCS never sits there waiting for the data, and DCS only ever has to open a small file.

 

In the long term, I very much want to switch over to passage-through-memory-only approach, but I just can't see how this is possible without ED making some changes. ED has the mission environment and especially the new scripting environment very heavily locked down, and short of seeing if there's a way of getting a .dll decompiler and editing the game's source code, I don't know if it's possible to change this (and I'm pretty sure ED would be opposed to this approach). The only way that I know of that the mission environment is able to modify data anywhere outside of itself is with print().

 

Unless someone can find a way to go through memory that ED approves of, then this is the only option left unless the day comes that ED gives us some data passing functions.

 

As far as not being workable, well, it does work. It's not like the functions are constantly passing data between environments. For all of the really calculation intensive functions I plan to create, they pretty much only run ever in a single environment.

 

So in the end, while a passage through memory is vastly preferential, the goals of the mod are achieved if:

1) It works

2) It doesn't slow DCS down

3) It doesn't compromise security

 

Those are the only things that will matter to folks who are using it in mission building and/or flying with it. They aren't going to care how ugly it is from a programmer's point of view.

 

And finally, it is worth remembering that it only has to installed on the machine of whoever is hosting the mission.

Edited July 27, 2011 by Speed

[Druid_]

Hats off to you Speed for persevering with this one. Although it does seem like a lot of work for something that might change with the release of the next patch.

 

If ED want to encourage Mods & a different approach to mission design then I really hope they are considering providing a safe & efficient environment in which it can be done easily. Once it is done then modders will not be put off by continuously having to change their Mods after each patch.

 

For instance a DCS function that allows us to write to a specific DCS folder only would be nice.

 

(My latest mission that allowed Game Loads & Saves is nerfed until something else comes out, maybe your workaround will help or maybe I'll wait a while & see what develops).

[Speed]

 

(My latest mission that allowed Game Loads & Saves is nerfed until something else comes out, maybe your workaround will help or maybe I'll wait a while & see what develops).

 

Maybe sometime you can send me the code and I can figure out how it works, generalize it into a function (or a Lua class if I've figured out how to do OO programming in Lua by that point), and include it in a version of the new mod.

[Speed]

So I took some time off, and decided to try to fix Bahger's mission "full frontal". It would be an easy task, right? Just replace the os.time() funciton with a random seed of my own making (I had to use nine triggers just to replace this one Lua call).

 

Turns out, the new triggered action script environment is even worse than I thought. Not only did they forget to give us print , but you can't even set a flag:

 

00068.844 ERROR Lua::Config: Lua error [string "-- Change the values in the 2 variables low..."]:40: attempt to index global 'trigger' (a nil value) in "-- Change the values in the 2 variables lowest_flag and highest_flag to the flag values you wish to use in your mission "

 

:cry:

So yea....... we need trigger.setUserFlag back too, please. Oh and some kind of os.time would be useful too. If they restore it there, then at least this new lua environment will be useful for some scripts.

 

Anyway, I did begin work on the Lua request post... but as I said, it's a long post. I'm not done yet.

Edited July 28, 2011 by Speed

[Druid_]

Oo thats screws all my missions up then (haven't tested any of them since I am down to one hand after shoulder surgery).

 

Why, why,why would they remove trigger.setUserFlag() ?!!

 

Feeling just a little frustrated by all of these changes. Maybe ED can tell us in advance of ALL the upcoming changes for patch .1.10 ? and even 1.11 while they're at it.

 

Ok so rant over, but rather than make us wait 10 weeks for the next patch how about a Hotfix to address the issues in this thread? I am sure there would be a lot of community members who would very much appreciate it, me included.

 

The alternative is that I for one will give up on mission building. I enjoy doing it & hopefully doing it well but I don't have the time or the inclination to keep going back & changing missions that are broken due to patch changes. So please ED help us out here.