Scripting in multithreaded DCS

[Guest]

Is my understanding of the following about scripting for multithreaded DCS missions correct? Or what is the lack of consideration?

1. Since each thread is created per task (i.e. per group), we need to add synchronization to access shared mutable resources across groups.
2. We can create any user thread we want. For example: a thread that controls the order in which groups are created when creating multiple groups at the same time with threads created for each group; a thread that aggregate the results of each group.
3. For example, when a missile fired from a unit in a group running in thread A destroys a unit in a group running in thread B, the unit destruction event(hit, kill, dead...) occurs in threads A and B respectively. Similarly, if a unit running in thread C refuels from a tanker running in thread D, refueling start and stop events occur in threads C and D respectively.

Event handlers run in the same thread as the initiator.
in short:

• A missile fired from a unit in the group running in thread A destroys a unit in the group running in thread B, Hit and Kill events are executed on Thread A and Dead event is executed on Thread B.
• If a group of units running in thread C refuel from a tanker running in thread D, the Refueling start/stop events run in thread C.

Edited March 12, 2023 by Fuujin
Correction of misunderstanding

[cfrag]

Well, if the Lua environment itself runs as a single thread, and doesn't provide tools to launch threads in the same environment, all this is moot. 

A likely MT approach to DCS would be to have many threads spawned to calculate each unit's next physical state. Any shared memory would either break that or cause dependencies so its likely that such dependencies happen in a second 'common stage' after all individual threads have ended. In such an environment, I think it highly unlikely that ED will open that up to mission designers, at least not in the initial stages, and hopefully only after designing an appropriate API. 

15 hours ago, Fuujin said:

A missile fired from a unit in the group running in thread A destroys a unit in the group running in thread B

This is not how MT usually is designed: objects commonly do not retain the thread that they are spawned in. Usually, objects are in a pool, and a number of 'worker' threads (depending on the number of available cores) are created. A dispatcher creates tasks that calculates the state for the next (and only next) frame for each individual object and assigns each task to the first free worker core (thread). A core (worker) takes the task, works through it, and then, after finishing, tells the dispatcher that it's free again. That way, a MT system can scale to the number of cores per system dynamically.

If then a thread that handles object A (a unit or group) determines that a new object is created (a missile), that object is allocated and put in the same object pool that all objects reside in. In the next frame's iteration that missile is worked on by whatever core becomes free while the dispatcher assigns tasks to free cores. Later, if a worker determines that this object (missile) has hit another object, it sets this as one of the previously determined results (i.e. there is no shared memory accessed). All result codes from all worker threads are usually processed when all worker threads have finished in a 'cleanup' sweep that resolves damage, kills, etc.

So objects aren't assigned to individual threads; the app creates a number of 'worker' threads and then - each time a new frame is calculated - distributes the currently available objects to the many workers iteratively (meaning: if we have 10 worker cores and 50 objects, each worker will work on 5 objects, one after the other, handed out by the dispatcher when it reports that it has finished working on the last one) until all objects are complete and the new frame state is determined. So it is completely normal that over time a specific object is worked on by many different worker threads, a different worker each time a new frame is calculated, and each worker thread creates some output that is retained. Then, after all workers are done, a single 'cleanup' thread is run that combines the result information from individual threads so that these threads do not have to access shared memory (which causes loads of headache in concurrent processing). For example if the thread that worked on unit xx notes that it has taken 100 pounds of fuel from unit yy, the cleanup phase transfers 100 pounds from yy to xx. The cleanup phase also would handle all damage processing and determines if objects are dead.

[EDIT: if you are interested, a good description of a parallel process dispatcher as implemented in Unix (MacOS, Apache) can be found here]

Edited March 12, 2023 by cfrag

[cfrag]

12 hours ago, Fuujin said:

Event handlers run in the same thread as the initiator.

[apologies, life intervened and I had to post and continue now]

As a result from the above, I suspect that all our Lua code will be invoked from one (always same) single thread, after the cleanup phase. All events happen in the same single 'mission' thread, and after all unit processing has happened. I doubt that for us mission designers (at least initially) there will be any concurrency requirements nor options. We must not assume anything with regards to the execution order (just like today) unless we ensure that ourselves, and that's pretty much it. We can assume that out Lua code is the only user-scripted code that is executing at any given time, and hence any race conditions result from our own code. 

12 hours ago, Fuujin said:

If a group of units running in thread C refuel from a tanker running in thread D, the Refueling start/stop events run in thread C.

I believe all events that we are notified of will run in the same thread. Anything else would raise scripting complexity by orders of magnitude and require ED to provide us with control structures (semaphores etc) that will make writing mission code an even greater hell than it is today, and completely destroy backward compatibility. I don't think that will happen.

 

Edited March 12, 2023 by cfrag

[Guest]

cfrag, thanks for replying with detailed explanation of multithreading.

My understanding is below:
1. Current DCS tasks are executed in an indeterminate order and always on one core. Multithreading allows tasks to be dispatched in undetermined order and to multiple cores for parallel processing. This is the point improved by multithreading.
2. Even if DCS object state updates are executed concurrently, user event handlers are executed within a single thread in the order in which DCS object state updates are completed and cleaned up (as they are now, their order cannot be assumed). So, as always, we mission creators don't need to learn multithreading to synchronize access to data we use in Lua scripts.

This means that Lua scripts for larger and more complex missions can become bottlenecks and slow overall processing time.

However, when I checked the Lua reference manual again, as cfrag originally stated, Lua certainly does not provide a means to implement a full-fledged multitasking program.
So does that mean we can't create large and complex missions and campaigns in DCS forever?

So what does a dynamic campaign engine look like? ? ?

[markom]

Note that multitasking is not really the same as multithreading. Same way as multithreading is not automatically concurrent, or parallel, or asynchronous, or synchronous. These are all possibilities that having multiple threads open up, but it's up to the implementation to do one, or multiple of these. 

I spent 0 time trying to figure out what kind of implementation DCS went with, and what it means for user scripts. However, it is entirely plausible (and also implausible) that each user-supplied script may get scheduled on a different thread than another user-supplied script. It could be that all user scripts get scheduled on the same core. These are the levels of details only ED can answer at the moment.

Note though, that programming for concurrent execution is complex. For example, imagine a simple data structure that has some variables and methods (aka: an object):

 

type person struct {
    name string
}

func (p *person) GetName() string {
  return p.name
}

func (p *person) SetName(n string) {
  p.name = n
}

 

This is Go definition of a structure defining person with one variable, one "setter" and one "getter" method. Now, this looks like a perfectly fine code.  However, this code is rather problematic in multithreaded environment. What happens when two threads call GetName and SetName at the same time?

This same class of problems will rear its ugly head if parallelized scripting is implemented in DCS... It's arguably a lot worse than scripts being bottlenecks :-).

Edited March 12, 2023 by markom

[Guest]

Synchronizing shared data is not too difficult a problem for me.
Synchronization is easy to implement using semaphores, Python's lock objects, and Java's synchronized block and modifier.
The only problem is that Lua doesn't have them.
Since Lua can call functions written in C, I think it would be good if we could implement synchronization and thread operation functions in C and call them from Lua.

[markom]

I didn't say it's an insolvable problem, just that it's a thing that exists as a challenge in even the most trivial code

While I am by no measure a Lua expert, quick binging for "Lua mutex" yielded some promising results...

[Guest]

markom, thanks for all your advice.
I know that Lua can implement non-preemptive multithreading and synchronization using coroutine.
However, I wasn't sure if it would work from a Lua event handler (if it was possible) that was called back from the thread handling the DCS task.
I could use it to do my own asynchronous processing in my Lua scripts, but it just complicates the implementation, and I thought it would be no different than recursively calling scheduleFunction and monitoring it.

[cfrag]

(to not cause undue confusion, I will try to use 'concurrent' processes/programming instead of multithreading/-processing/-tasking. I'm talking about a system where more than one thing can happen at the same time without looking at the underlying architectural details)

I believe that in the foreseeable future, "DCS MT" can dramatically accelerate an individual unit's next state calculation inside the game engine (this can be vectorized similar to rendering the scene), maybe in a manner described above. I think that - if DCS follows the model of other game engines - the phase where user scripts run is a single-thread phase. Since today all DCS objects are immutable to Lua, this is no change to before and is backward compatible. This may indeed represent a potential bottleneck, but DCS may still "cash in" on the performance gain during individual unit processing, and it doesn't represent a worsening of today's situation when executing Lua. Perhaps add a JIT Compiler and all is great

That being said, the ability to easily (or worse: unwittingly) create concurrent code is not something that should be unleashed on the average mission designer (example: each trigger can invoke its own thread and all Lua scripts invoked there run on a different thread. That would be Hell, capital "H"). Today, a single endless loop can freeze DCS. Imagine that you now add the joys of deadlocks and race conditions (which mutex and semaphores can't prevent) to the mixture by introducing unmanaged concurrency. Creating working concurrent code is hard, and requires a stellar architecture (currently definitely not in place) and an impeccable understanding of the underlying technologies and limitations that very few mission designer (should be required to) have. Write a compute shader (heck, just any common GL shader suffices as an example) and we get an inkling of the challenges involved - and that is in a tightly controlled environment: all we have to care about are some mild-mannered IO arrays. Requiring mission designers to also contemplate concurrency issues could straight out kill their enthusiasm and (considering the dearth of apt mission debugging tools) make it impossible to write non-trivial missions. So - at least in the beginning - I think it would be a good idea to limit access to concurrency for the average mission designer.

I think that over time there *will* be mission tools that enable concurrency, perhaps even separate concurrent environments, and hopefully adequate development tools. In the mean time, mission designers like us should keep an eye out for performant code and good runtime behavior while continuing to exist in a single-thread universe. 

Edited March 13, 2023 by cfrag

[cfrag]

14 hours ago, Fuujin said:

So does that mean we can't create large and complex missions and campaigns in DCS forever?

I think we are conflating 'complex' and 'large' with bad performing, while at the same time assuming that concurrency may resolve such an issue. A large/complex mission may be large/complex because of many units. DCS MT may resolve that easily today inside the game engine.

If, on the other hand, your code is breaking performance because it iterates all units every frame (everything everywhere all at once?), or worse exhibits an exponential runtime behavior, then even MT will only help you marginally. The issue isn't the number of units per se, but a suboptimal stretch of code. Our mission code should not be a drag on performance, and I have seldom seen bad performance resulting from badly written code being resolved by concurrency. Also, there are very few people who can reliably create complex concurrent code, making concurrent mission scripting the absolute exception. So even if the mission's Lua thread is a single one, that should still allow large complex missions. Mission code usually is control code, not worker code. There will of course always be good example to the contrary - but I think that single-threaded mission Lua is not going to be a show-stop for large missions. On the contrary - given the fact that so few people have mastered the art of concurrent programming, I think it's a necessity that all DCS mission scripting is single-thread.

Edited March 13, 2023 by cfrag

[Guest]

It's funny to say that DCS mission scripting needs to be single-thread because most people can't concurrent programming.
Only people who can use multithreading should be able to use it.
It just provides API for multithreading.

The large, complex missions I want to create aren't just missions with lots of objects scheduled in the mission editor.
Trying to design a realistic mission would require a lot of coding.

However, after looking at some campaigns and missions (mostly free ones), most campaigns focus on gameplay and none of them have complex implementations.
If no one wants to create a realistic mission, I don't think anyone can imagine the mission I'm thinking of.

However, if there is a campaign that allows a more realistic operation even if it is paid, please let me know.
Perhaps I can't know what implementation it has. But I may get a chance to improve my implementation.

By the way, when trying to figure out the position of F-14's gauges and switches in Lua code, there's nothing I can get them with the Unit:getDrawArgumentValue method. So I had to set the trigger rule in the mission editor to get them. Currently, of course, all triggers are executed in order, so the more repetitive triggers, the slower the whole process.
I hope that triggers can also be processed in parallel by multithreading.

[Guest]

If you want to know what kind of mission I want to create, read "F-14 Tomcat Units of Operation Enduring Freedom" by Tony Holmes.

[TEMPEST.114]

I think people are way over-thinking this. 

Lua has coroutines if you need to do some simple offloading, but it's just a scripting language - it's backed by a C/C++ class that does all the real work, so leave it to ED and those who manage that C/C++ stuff to handle how Lua should be threaded when it does it's work. 

You really don't need to think of threading in Lua - it's just script.