DCS Multi-Threading Status Report February 2020

[Worrazen]

So here we are, after quite a large gap between the last time I was doing diags with DCS, I kinda put that deep one on hold because I didn't want to jump to any conclusions, with a program a lot more detailed and sophisticated than the one in this post. I wanted to get in to the bottom of what's responsible for what, but I realized the task is quite a bit more complicated.

 

With quite some new people around since then, who aren't up to speed with the DCS performance stuff, I decided to make this overview of the DCS threads behavior, with some general observations and not trying to go into definitive detail, it should do the trick for most people to get a sense of the status of "Multi-Core Support" (there's no such thing :p)

 

Initially in this post, I got into a small A-10C mission, and because I've seen different behavior with different modules, maps, types of units on ground and/or air, radars, EWR, the types of actions you're performing (TGP usage, types of weapons), and so on, it is very important to always associate any such DCS performance stats with a specific scenario that was used when recording.

 

This case obviously did not have any missle AA units, scanners, radars, EWR, there was no ECM usage, no countermeasures, and I didn't even have GBU's to drop due to mission's default loadout choice.

 

 

The way you read/interpret this PerfMon graph is, .... well I'm not really sure as I used it more lightly, which is weird as It's me who should have figured out this before posting, but this is all niche stuff with scarse info on the web, the point is it's good enough to get you an idea how things may be under the hood.

It probably safe to say that 1 thread isn't taking, at some point over 80% of the whole CPU which is a Quad-Core, that wouldn't make any sense, so that wouldn't be correct interpretation.

One thread can only do 25% on a Quad-Core with HT disabled, so 100% of all cores divided by 4 is 25%. A lot of statistical data is lost when viewing total CPU usage of a process, that's why using Task Manager is not recommended, it's hardly reality at that point, most system administrators and other IT specialists within the industry know that Task Manager is garbage and it's solely made for the average user to get a rough idea of the system's health and status.

So interpreting this graph in a sense that when you look at a color, a thread, the scale on the left being how much this thread was taking in regards to the CPU core it was on. So Blue Thread was taking 80% of CPU Time on CPU2.

That is the correct idea and does apply in general how it works, but still take this specific graph with a grain of salt.

 

An example of a finding:

 

(There's so many threads that one got the same color, I forgot to change one of the reds to a different color)

 

So we got mostly Blue, Two Reds and Cyan, and spikes from a Yellow. So everything more than 4 is going to start filling into the same 4 CPUs we have in this case, but keep in mind that we're not maxing out the other 2-3 cores, there's still space for many threads with light load to fit into the 3 threads.

 

Now you can see where this case would get a potential slowdown, or a stutter, you can see in the middle roughly there's a spike of a Yellow graph, and that's a 5-th color, that would mean we got one thread here that has to fit in those 4 Cores and potentially be forced-in taking away headroom from the rest, we might be bottlenecking, well not necessairly ...

 

... Remember we still have headroom, that Yellow spike shouldn't affect the main thread, it's not big enoguh, it could fit in any of the cores that still have headroom, but what we can observe suspiciously is that there is a correlation with the main thread dropping down at the same time the Yellow thread is spiking, that's not what we would like to see if we want optimization, it could be a coincidence or a bug/issue/design in the DCS engine it self.

Having to pause some type of workload on the main thread while a different workload on a different thread is taking place is extinguishing the point of multi-threading.

Such cases, if true, would give us a false sense of optimization and multi-threading, if we were to just look at raw numbers, eye balling the data, looking at number of threads and their activities. So with all diagnostics and analysis it's very important to look at many different types of views and take things seriously, interpreting the diagnostic data correctly is very important and the last thing I would like to do here is to create some panic among the community by wrongly interpreting the data, or giving false sense of performance. What I can say for certain it's definitely better than what people think it is.

 

Back to our little example here, ... if the main thread has to stop for another thread to work then what's the point of that workload being on another thread right! But we still have to take all the factors and conditions into account, this graph is a replay of a 30 minute A-10C flying around doing stuff that you can see in the video, but also AIs doing it's own thing, it can get very complicated of who's responsible for what spike/event on the graph, you can separate with specific scenarios designed for that, but in general gameplay it's all molded together. I've did many such things in the past and boy the graphs get interesting when you destroy 500 units and there's a huge load of smoke, those are specific case examples which are complicated and I didn't want to post those on purpose because it's hard to explain all the tiny circumstances, without doing a live take video where you would see, unfortunately the other programs are extremely resource intensive and wouldn't be able to run both DCS and them at the same time, recording diagnostic data is already a resource hog by it self, but Windows Performance Toolkit isn't even designed for real-time operation anyway. So that's why it's key to not jump to conclusions, and the deeper you dig it actually presents more questions and more places to dig further, then it's obvious that everything's extremely relative to the scenario. People that report stuttering versus other people that don't may just be using different modules with different maps playing a different style of gameplay and using different weapons and systems.

 

So when looking at diagnostic data and graphs always keep a sharp lookout on the whole situation. Again, this scenario was not using any radars, any ships, and missle AA batteries, all the data is relative to this scenario in the video below.

 

There may be other benefits of putting things on a different thread but still being kinda tied to the main operation from various development point of views that I do not know about, but maybe that's a dud, I don't know, but generally something happening on another thread should not be stopping other things and getting in-the-way, that said this is still looking back at DCS as most of the core improvements haven't come, it's a look back at the last decade, so if anyone's reading this years down the line, there's no reason to be concerned, it's looking far better as it is, we all thought it's only running on 1 thread, well nope, so things are just going to get better from here on out.

 

There's no reason to concern overall, there's no cases where it's super bad, overall DCS is hard on the system yes, but I didn't see any hard bugs or indications of them, except the A10C_Protect.dll and FC3_Protect.dll seem to be kinda legacy things chugging along there taking a bit of CPU regularly, even tho if you're not flying A10C or any FC3 aircraft in a session, again those may just be names with other modules using it, while I'm speculating other newer modules seem to have their protection performance hit probably tied with DCS_protect.dll which isn't that big alone, let's just hope it's not stopping the main thread when it's taking CPU time.

 

 

A basic overview of the threads with approximated descriptions:

 

 

 

The basic case scenario video:

 

 

 

So what are the observations we can probably make:

 

• There's one thread that seems to be taking more CPU time than any other one.
  
• Audio seems to be on it's own threads
  
• Storage Access (IO) seems to be on it's own threads
  
• Copy Protections seem to be on their own threads
  
• Some part of the graphics and driver stuff is separate
  
• Some controls input may be separate as well
  
• TacView plugin is on it's own thread too
  

 

Also, please note that the thread Start Addresses may not be 100% accurate/valid of what we're assuming the workload is, because it can be convoluted in code as things may be called from one place but work being done elsewhere, spread across differently below the calling piece of code. So don't take it as it's written in stone, althought it does make sense with the rest of the data for the most part, we know from the devs that IO is split from the main thread, we know from the devs that Audio is split from the main thread, at least that's what I'm aware of.

Just keep in mind that there could be other types of work on that thread apart from what it's Start Address may seem to tell us about, so an audio thread may not be exclusively doing just audio work, for example.

 

 

2020 And Beyond: (More opinion style!!!)

 

We'll see a major change in these stats when Vulkan API comes, unless some big new feature comes that takes up all that gained performance, but I doubt it, I believe all the driver overhead and draw calls run on the main thread and that's a big thing Vulkan based GFX engine will deal with.

 

The next big thing to alleviate off the Main ("simulation") thread is the AI if we are to believe the basic thread info, it has to be spun off the main thread ASAP, I don't see how AI logic per-unit would need to be on the main thread where physics calculations happen. There's proably loads of serial workloads that don't have anything to do with each other math-wise. While you fire a missile at an AI1 airplane and that missile calculations have to go on and it's taking a lot of CPU time on Thread1, what's keeping some distant AI3 from running it's separate AI logic on Thread2 for landing or whatever, right right!?!? So there's a lot of potential to split away these serial task into many threads. Each unit on the battlefield or rather squad could probably run not only in it's own thread, but it's own set of threads (by type) if you ask me.

 

Mission Editor logic and triggers, waypoints, etc, maybe don't have to run on the main thread either (I could be wrong a bit if there's a thing to do with events/detection, but conditions nah) So if the ME logic, a condition, checks for IF THIS UNIT IN ZONE A, it doesn't have to be on the main thread, sync with physics shouldn't be needed because it may fire what at most 100 ms later than it would otherwise, that's probably nothing in practice, and DCS matches with complex ME logic will probably have 1000x of factors so the chances of this discrepancy, but then again I'm talking worst case scenario. What I'm talking about is like, you entered zone A by a few meters, but because ME logic was late it did't register, but then you got hit by missile and then you happen to fail the mission. Chances of this being a problem? Then again, because the Main thread is bloated, moving it off might even make things better for this part. The thing I'm afraid is that this sync could be important for Multiplayer and you don't want each client have different away from what the main network is sending out, ... but what the heck am I saying, ME Logics should and probably is running exclusively on the dedicated server host, not the clients.

 

So we'll see how deep the devs are willing to go to split what is splittable, it's a big deal for engine programmers though and not something that will happen overnight, they should really hire some extra IMO because the longterm benefits are worth it especially when there's features coming that will need that extra performance headroom, but also new CPUs are going crazy on amount of cores.

 

Once this is all done, the Main thread may look less and less like the main one in terms of CPU Time, that's right, but you're still suppose to call it the Main one IMO, because in technical detail it is still the one holding the process together, the engine, being the parent and responsible and stuff ;)

Edited February 12, 2020 by Worrazen
typos

[Worrazen]

Here's another case with F-18C EarlyAccess CAS Mission (Forgot to choose better loadout)

 

Compared to the initial case, this mission has far more things going on, there's far more AIs, ships, carrier, AWACs, Kub AA with radars, the proprotion of the main thread vs the rest gets wider as you can see.

 

 

 

Edited February 11, 2020 by Worrazen

[112th_Rossi]

Threading is not a magic bullet for performance gains. Also, CORES != Threads! They are not the same. You can have a multi threading application on a single core CPU with context switching.

 

There's a ton of stuff in game that must be kept in sync. So pushing off a process to another thread only for it have wait until another process has caught up to remain in sync negates the point of multi threading an application. However a few things come to mind when deciding what process could potentially be asynchronous

 

Weather,

Audio

ATC communications

Network packet processing in multiplayer

UI elements

Special effect rendering

 

Items that do not necessarily need to be kept exactly in sync with the main game loop in order to work.

 

I wish people would stop thinking that threading is something that will solve everyones performance problems. It just wont.

[SkipCarey]

So Vulkan isnt the holy grail? from what Ive read (Wiki) we can expect a 20% in crease in performance at best from Vulkan @best

[112th_Rossi]

So Vulkan isnt the holy grail? from what Ive read (Wiki) we can expect a 20% in crease in performance at best from Vulkan @best

 

You think you can port over DCS and magically get 20% better performance?

[deathbysybian]

Ai craft, vehicles, scripts, projectiles could all be distinct threads.

[sk000tch]

You think you can port over DCS and magically get 20% better performance?

 

Well gosh that would seem to be a absurdly generalized statement with a few assumptions...

 

What does 20% performance even mean? Frames per second? I realize everyone seems to think CPU is their bottleneck, but DCS is extremely costly to render per pixel. VR is worse, requiring a huge amount of pixels to be rendered to have a cockpit with readable gauges and MFDs.

 

Vulcan's big pitch might be its multithread support but would you be surprised to know multithreading and vulkan are separate projects?

 

I bet there might even be some benefit about not being on proprietary APIs? Ever see those driver updates that get released for AAA titles? Think MS or Nvidia answer the phone when the little russian 100 person dev with the niche flight sim calls? What if DCS is thinking down the road a bit and is planning mutli-GPU support, single pass stereo rendering, VRS support for foveated rendering, shader optimization that doesn't require brute force MSAA to make it look non-ass like? That would be better right?

 

Makes you wonder why training sims that prioritize fidelity simulate flight throughout envelop very accurately, including simulating departures and other scenarios that are unsafe IRL, run on dual CPU behemoths? Fluids man, they do crazy shit. I wonder if ED has had to make sacrifices due to decreased CPU resources on consumer hardware?

 

So if after vulkan and multi-threading is done, and implementation has matured, my plane flies better and has more accurate damage modelling? And the AI is better? Is that performance?

 

Not trying to pick on you dude but none of this is news, they got something like half the dev team working on this and will be among the first sims on DX12 or Vulkan. Maybe MS beats them to it. Fair enough, they've got a few more resources. Maybe the reason is they sit on a 99% build for months to avoid these complaints. Low level APIs can be great as we've seen but do not reward poor implementations. It's no magic bullet and will probably be worse at first. Expect parallel builds.

 

Maybe, just maybe, we should put away the pitchforks before we see what they pull off. Everytime this topic comes up it's just painful. At least nobody has posted a task manager/resource monitor pic yet, so there's that i suppose.

[112th_Rossi]

I know that was my point hence the 'magically'. I guess sarcasm is lost in forums

[BIGNEWY]

Hi all

 

by all means discuss this topic, but please be respectful to each other.

 

Thanks

[sk000tch]

I know that was my point hence the 'magically'. I guess sarcasm is lost in forums

 

Sorry man I wasn't intending to pick on you. The comment, even in sarcasm, was just perfect to quote upon which to vent my irritation (with a healthy dose of sarcasm as well). Even though I said it a couple times I should have been more clear about that as it came across more directed at you than was my intention.

 

No worries bignewy, everybody is getting along. How are the projects going anyway?

[Worrazen]

There's a ton of stuff in game that must be kept in sync.

 

There's a bunch of stuff I don't want to get into details, and I remained light in this thread, I won't go into deep speculation so much as I was trying to in the past, because I really don't know and it's tricky, it's quite a stretch to guess how this simulation sync could matter, where could it matter and where not, and how much, do two missiles that fire from different unit and are tracking a different target have to be synced so the physics/tracking calculations all have to be on the same thread ???

 

There's like a paragraph I just wrote and deleted, it's pointless, we don't really know, unless you're with 10+ years of experience in deep CPU programing, geometry layer, physics, tracking logic, and 100x other things, what's splittable, could potentially all weapons be on their own thread, like you drop a GBU-12 and it works in thread20 and your buddy fires an Aim-9X and it works on thread21, or at least partially, ... as an example, now in practice ofcourse these things wouldn't fill up a core so in practice these relatively small split workloads could be grouped into a thread so that there wouldn't be thread spam with so many threads doing tiny work.

 

I got to say that with a big note that this is all at your own risk information, I kinda wanted the light and the big plausible details to be known out there, without loads of my speculation which can be negative I mean it's like a stress generator and probably quite a demotivator and I don't want it to take the attention away from the actual content and the reason we're playing and being here and there's a lot to be happy about.

 

I'm actually learning the tools and figuring out all the other side stuff around them, as I'm writing a custom program for anaylsis of the processed output of these perfromance loggers (EWT) for an unrelated use case in performance analysis, as I deal with diagnostics/perf for a number of other stuff, so that's where the root causes are, DCS is more like of an example test case I'm using to get to what I'm actually after, so yeah it's less about purposelly hunting around DCS.

 

Going forward if I get deeper into this I'll probably do posts that would be more geared potentially to actually be helpful as a bug report, or if not at least interesting to the tech savvy part of the community here, trying to teach every single person is a bit too much task but I think it's rather helpful to just do various scenarios and get the data out and report and leave the speculation out about what we or me or she or he would do about it and all that stuff.

Edited February 14, 2020 by Worrazen

[112th_Rossi]

Writing an application that can use parallelism effectively requires an upfront design decision from the beginning, before even a line of code is written. A multi-threaded app, architecturally is very different from a single threaded one. So we are all just going to have to get used to the fact that the current iteration of DCS is simply not going to use all the power our precious CPUs can give out.

 

I'm sure the dev team are fully aware of this and I'm sure they will want to do something to take advantage of new CPU architecture. I just don't think posts like this are helpful to anyone since it will not change what currently have and simply states the obvious.

[Worrazen]

Agreed but with this specific I purposelly tried to give the new people some insight because I saw some posts about "multi-core support" as if it's a feature you slap on, as I noticed it did relatively good on reddit there. The semantics is a big deal with me and it springs me into action sort of, actually I was already doing work for an unrelated project with these tools so it was just a quick jump, I totally get it, I had reservations my self should I keep punding on this issue, but I decided if it's a good approach to this then it should work out.

 

Besides on a bigger scale, it at least keeps the less patient players talking about something while they wait for those juicy core upgrades, that's probably better than being bored right ;)

 

That's exactly why I'm going to do going forward for the most part, that different kind of posting which may potentially help in the current state, and I'll keep those thread more low-profile so it doesn't unnecessairly take away from the main point.

I'm thinking in terms of finding some areas where there's is a feeling of excessive CPU usage in a particular scenario, finding such bits that may infact be regressions which worked properly in the beginning, finding such bugs, and other inconsistencies, which could count as a good bug report and perhaps actually be fixable as per standard, without any of the new multi-threading stuff having to be speculated around and around, that's probably is helpful would you not agree?

Ofcourse the devs may know all this more or less if they dig in with their own internal tools and they may have ofcourse a way better overview and perhaps custom/enterprise tools to track performance, on the other hand it may not be the case because DCS is big and I heard many times that there's always busy times and they encourage bug reporting.

Edited February 14, 2020 by Worrazen

[grammaton_feather]

Shifting to Vulkan and with a multi-threading shift the performance gain will be significant. It's not necessarily just about increasing FPS though as with that other flightsim upgraded to Vulkan... a big boost in smoothness and a big reduction in stutters and frame drops.

 

If DCS switches to Vulkan but the code continues to load a single core then people will still need to push that core close to 5ghz. As DCS stands right now for VR use you'd really need a CPU core at 10ghz to push up the FPS to the point where you're not just relying on ASW or other VR driver motion-smoothing tricks.

[Worrazen]

Even tho there's multiple threads for I/O ... it's not completely watertight, while I'm determining if it's due to thread-jamming (multiple threads racing to get CPU time and the main one get's unlucky for a moment) which can cause stutter or it's really due to some part of the asset loading process (texture streaming, model, effects, loading) ... the third option is the latest one which is shader compiling.

 

There's a particular laggy just when you fire a rocket with F-18C for the first time since starting up DCS, but I had my shaders cleared in that case, I've seen it happen before but didn't record it for documentation.

 

I have figured enough now to realize what I have to do, I need to start logging and documenting and also videotaping all the stutters that happen at the same exact time there's a fetch and read of DCS data from disk.

 

It may take some time tho, I'm in the middle of a large PC/HDD/data maintenance and I'm looking into testing another version of windows to move past 1607, well I probably won't move completely, probably 2 or 3 concurrently, but I'll put DCS on the newer version together with some other things I plan to have there earlier than originally planned, running out of HDDs/SSDs to put new Windows onto dammit.