[INVESTIGATING]Too many Disk I/O threads saturating the CPU/Main thread? (WORKAROUND)

[Worrazen]

-----------------------------

Original Post from 2020:

[WIP - OP from 2020 was reworded for more clarity, readability, less focus on the side issue, formatting and coloring for new forum system and compatability with further updates - Feb 2021]

 

Hello

 

Who would have thought that would ever come to a point like too much multi-threading ... unfortunately it may be so in the worlds of computers. With all the multi-threading buzz and anticipation sorrounding DCS World, this is a good example to show that multi-threading has it's particularities as well and optimizations that have to be finely tuned.

 

I have decided to take the time and treat the community with a bit of a commentary, so this is a multi-faced forum topic that is more than just a bug report. The key ofcourse is the data for the team, but I went a step further with the explanations for the curious folks, including a good reminder for my self when and if I come back to this issue, or a similar scenario unrelated to DCS, it happens. For the average players who aren't interested it may be best to avoid most of it, if all they wish is a TLDR. While a TLDR is possible, that isn't the goal with these sorts of reports.

 

Intro - includes test run specifics and background

 

This is my understanding at this point in regards to what's happening here with DCS and anyone's welcome to correct me if they feel like I'm missing something or coming to a wrong conclusion.

 

I tried to get a real take on a different machine for this one, to make extra sure it's replicated more broadly rather than on just one PC configuration. DXDIAG of both is pasted below, unfortunately it wasn't what I was hoping for, the other remote machine didn't quite show enough of this workaround working, or I made a mistake when testing, I also think it could have been due to other reasons:

 

• The CPU is weaker and it may not be able to accomodate the Main Thread's increased activity when large portions are being streamed and other things during very active scenes, even if it had a whole core for itself.
• It has only 16 GB RAM, however it feels that DCS did notice and adapted to that, still that means the pool size is lower and it has to perform more Disk I/O as it's trying to keep a tighter RAM budget, which is correct how the proper dynamic asset loading systems should work, except in DCS when something has to be loaded it takes so much CPU (it may be graphics/rendering related too but not going deeper on that right now)

 

I did not record anything while testing on the secondary PC, the weaker CPU there makes that difficult to do simultaneously while trying to not interfere with the test.

 

It may not be the amount of threads themselfs, but just because the combination of my amount of CPU cores and enough other threads that saturate all of them, people with much higher core counts may not see this issue nor require this workaround.

 

Continuing, perhaps the default thread priority may also have something to do with it, if it's not optimal, adjusting that may help and this may be where the room for improvement could be. I haven't tried testing thread priorities in this run.

 

The mission I used in this session/run, the numbers and claims are referring to, is a barebone flight on Caucasus map. I picked the term "barebone" meaning that it's a solo pilot plus an empty airplane free flight without any other activity on the map, the exception to the barebone is the use of TGP and MAV which is noted, so "Barebone+TGP+MAV" for example.

 

I have mentioned for a couple of months around ED forums how I reached a sort of dead-end, because to do what I have foreseen back then, would require CPU affinity modification on a per-thread basis, without having to modify DCS files, is key memory, or otherwise interferring with the clean public client version, so that any possible workarounds or testing would be possible for the public at large without any kind of messing with the anti-cheat and authentication systems.

 

Turns out there is a way to do it already from the outside just like I hoped for with the WIN32 API, ofcourse developers at large who make software for Windows and use it's SDK's can do this easily and it's no big deal nor anything new.

 

In an amazing turn of events, I found out it's already been implemented for quite some time in a program that for some reason I never came across until a few days ago, well I must admit I didn't look hard enough, all the years I was into Windows tweaking, troubleshooting and optimizing. I guess the saying of "you never stop learning" is quite something that rings around us all the time.

 

Process Hacker can do just that, without interfering with DCS (AFAIK) and this opens up many ideas what I dreamed about with many kinds of programs/games in terms of perf diagnosis and stuff, but the bigger thing is, it opens up potential for workarounds and temporary solutions. In this forum topic I explain one such possible example.

 

 

--------------------------------------------------------------------------------------------------------

///////////////////////////////////////////////////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------------------------------

 

 

Overview of the threads in Process Hacker: (~Q2 2020)

 

 

If I would take some of the thread priorities at face value, I'd say they are suspicious and may not make sense but I don't want to make any conclusions whatsoever on that. I can see the DCS Main Thread is "Normal" while several of the data loading I/O threads are "Above Normal", given that we can agree on engine responsiveness being more important than a few textures coming in a bit late (miliseconds, etc), then those priorities should be the other way around, vice versa, if I understand what these priorities mean and do, which am not sure right now.

 

To remind ourselfs, dynamic texture loading is suppose to be down the list of priorities, because of what I think are players priorities, seeing a higher quality texture LOD pop-in a bit later (may not even be noticable difference) is probably much much lesser of a negative impact than experiencing a stutter or more.

 

Speed is welcome ofcourse, hence texture loader I/O is split into many threads to load many section of terrain textures and materials simultaneously, all without interfering with the gameplay, without needing to pause for another loading screen for example. Old retro games didn't need streaming back then because they just loaded everyhing in RAM in the beginning, at the loading screen section. Many modern games for past decade, while still having loading screens, are still too big to practically fit everything in RAM, they load things while you play and there's different kind of data streaming methods and implementations from game to game. They all have a common goal, populate the 3D world ASAP without affecting playability right, it should never interrupt the execution of the core engine, so that there are no stutters.

 

Reminder that a stutter in this case (seen as a FPS drop) is caused by the pause in execution of the main thread, let that be clear, it has nothing to do with GPU and graphics, FPS drops as a consequence because there's simply nothing to render for a period of time.

 

--------------------------------------------------------------------------------------------------------

///////////////////////////////////////////////////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------------------------------

 

The Test:

 

The most obvious scene when I think the Disk I/O loading has the biggest impact is with the use of a targeting pod and other imaging sensors that scan or view terrain far infront of the players aircraft, such as a TGP or MAV imaging. Those distant areas may have to be loaded from disk because they weren't loaded before, since RAM is limited, other data has to be dumped from RAM to accomodate the new terrain data where your sensor is looking at that moment, so the cycle can continue unless one has a really big amount of RAM and VRAM.

 

So for this test I used A-10C with TGP and MAV flying around the Caucasus map, as mentioned it's barebone, so no other units or activity of any kind. I only posted the results from my primary Win10 PC tests.

 

Default CPU Affinity Case:

 

Image Album: https://imgur.com/a/hzadyBf

 

 

 

 

 

 

--------------------------------------------------------------------------------------------------------

///////////////////////////////////////////////////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------------------------------

 

 

Workaround CPU Affinity Case:

In this case I used Process Hacker to set the DCS Main Thread's CPU affinity to only 1 CPU of the 4 I have (HTT/SMT disabled), while setting all of the other threads away from core CPU0 accordingly, as well as some of the key processes unrelated to DCS such as Process Hacker it self, MSI Afterburner, DWM and some other system processes that are critical but may take 2-3% of 4xCPU, so the DCS Main Thread effectively "gets" an exclusive core to it self.

 

 

 

The threads you see on the screenshot above are from the fully loaded mission start state, on the "FLY" screen.

 

Image Album: https://imgur.com/a/o8qvQ79

 

 

As you can see the behavior with the A-10C TGP is much better, while the FPS is expectedly variating, there is no hard stutters or very low FPS drops of any kind, it feels smooth, if I wouldn't have any graphs displayed and playing normally I wouldn't notice the FPS differences.

 

However, this can't be easily seen with just the screenshots, I avoided recording a video when doing these screenshots because the CPU requirements of real-time recording would interfere with the test results. It would require a much beefier CPU, or GPU that could handle the offloading of video encoding and the game at the same time.

 

--------------------------------------------------------------------------------------------------------

///////////////////////////////////////////////////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------------------------------

 

Side-Issue: The Loading Screen Unresponsiveness

 

 

I distinctly remember this did not happen some years ago. When you click with mouse the standard Windows unresponsive error appears, but it has no consequence, loading as normal. DCS may not properly be signaling Windows it's infact all okay, just loading something really hard.

 

This may just be a rather minor issue with the way app responds with windows, analysis shows that the main thread is 100% loaded during loading screens half the time, while disk I/O loaders also help out. Fixing this won't mean the main thread will be alleviated of much of the workload, we don't need it to.

 

--------------------------------------------------------------------------------------------------------

///////////////////////////////////////////////////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------------------------------

 

Testing the track yourself:
 

Spoiler

Track Attachement: https://forums.eagle.ru/attachment.php?attachmentid=230972&stc=1&d=1585447346

 

And you need to own the A-10C module ofcourse first.

 

If you wish you can follow this guide because it's good to replicate the MSI Afterburner / RSST overlay to be able to see and compare difference, but there's definitely more than one way doing it.

 

• Install MSI Afterburner which comes with Rivatuner Statistics Server (don't deselect in setup)
• Go to C:\Program Files (x86)\MSI Afterburner\Profiles and find MSIAfterburner.cfg
• If you already have MSI Afterburner installed, make sure it is closed, backup that .cfg file by creating a differently named copy, delete the originally named one and start MSI Afterburner so that it makes a new fresh default one.
• Close MSI afterburner again and open the fresh new MSIAfterburner.cfg for editing in your preferred text editor but prefer Notepad++ or similar.

 

Edit by adding or overwriting the following lines in the code box below. Do not copy the whole contents, you unfortunately have to do this manually because there's a bunch of automatically generated lines specific to the system and GPU/drivers you have, so I can't just give my own MSIAfterburner.cfg and expect it to work. Do not delete or change any other line unless it's specifically present below,... hopefully I didn't forgot something because one line/letter could break it possibly. Make sure the lines are placed in the correct "[GROUP]".

 

[settings]
Sources=+GPU temperature,+GPU usage,-FB usage,+Memory usage,-Core clock,-Memory clock,-Power,-Fan speed,-Fan tachometer,-CPU1 temperature,-CPU2 temperature,-CPU3 temperature,-CPU4 temperature,+CPU temperature,-CPU1 usage,-CPU2 usage,-CPU3 usage,-CPU4 usage,+CPU usage,-CPU1 clock,-CPU2 clock,-CPU3 clock,-CPU4 clock,-CPU clock,-CPU power,+RAM usage,+Commit charge,+Framerate,+Frametime,-Framerate Min,-Framerate Avg,-Framerate Max,-Framerate 1% Low,-Framerate 0.1% Low
HwPollPeriod=300
OSDToggleHotkey=0006004Fh
[source GPU temperature]
ShowInOSD=1
Group=GPU (your gpu short name) -> 
[source GPU usage]
ShowInOSD=1
Group=GPU (your gpu short name) -> 
[source Memory usage]
ShowInOSD=1
MaxLimit=8192 [DELETE THIS: your max VRAM on the GPU in MB]
Group=VRAM (Max: [your max VRAM on the GPU in MB] MB) ->  
Name=VRAM usage
[source CPU temperature]
ShowInOSD=1
Group=CPU (num-Core num-Thread) ->    
[source CPU usage]
ShowInOSD=1
Group=CPU (num-Core num-Thread) ->    
[source RAM usage]
ShowInOSD=1
MaxLimit=32768 [DELETE THIS: Installed RAM minus HW reserved]
MinLimit=0
Group=RAM (MAX: [installed RAM minus HW reserved] MB) ->   
[source Commit charge]
ShowInOSD=1
MaxLimit=40960 [DELETE THIS - your limit, usable RAM (installed minus HW reserved) plus pagefile]
Group=COMMITED (LIMIT: [your limit, usable RAM (installed minus HW reserved) plus pagefile] MB) ->                  
[source Framerate]
ShowInOSD=1
MaxLimit=80 [DELETE THIS - mine is slightly above monitor refresh rate 75Hz]
[source Frametime]
ShowInOSD=1
MaxLimit=100.0 [DELETE THIS - too high would make the graph look tiny but would make extreme lows really visible, 100 ms is 10FPS]
MinLimit=12.5 [DELETE THIS - 12.5ms equates to 80FPS in line with the FPS graph] 
[OSDLayout1]
GraphWidth=-52
GraphWidthEmbedded=-4
GraphHeight=-4
GraphMargin=1
GraphStyle=1
GraphLabel=3
GraphPlacement=2

 

 

I can't include everything because disk instance numbers are again system specific, you have to go to the GUI and set that up on your own from there. Instructions continue:

 

Save the modified file and run MSI Afterburner and go to the settings and Monitoring tab, once there follow the image guide below to enable disk counters:

 

 

Then once they appear in the left scroll, you only really need either usage or read rate, pick one or use both it's up to you, but not the write because DCS doesn't have anything to write into the install directory during gameplay or is most likely not relevant to what were testing here.

 

Make sure "Show in On-Screen Display" is selected and set to graph or both whatever you like, but keep in mind adding too much lines will fill up your horizontal space in the OSD and some things may be outside of the screen.

 

 

---------------------------------------------------------

---------------------------------------------------------

 

You can perhaps restart MSI or the whole PC right here to start off with a fresh memory state without any browsers and make sure nothing's using up the CPU, antiviruses should be temporairly disabled, and no background updates.

 

DCS Should be started right here, if you want to fiddle with the Loading Screen then don't load the mission before proceeding, otherwise load the mission/track fully but don't press FLY.Alt-Tab out of it (Alt-Tab isn't the proper way to minimize as per Windows goes, I know, but other ways don't work for me)

 

 

Then we open the Process Hacker, go to properties of the DCS.exe process, and proceed to the Threads tab. Skip this step when testing the default affinity case.

Once there we set the DCS Thread Affinity like so:

 

• The Main "DCS.exe!" thread to use only Core 0.
• All Nvidia/AMDATI/dxgi threads to use all other cores except Core 0.
• All windows-OS-looking threads except "ntdll" named ones to use all other cores except Core 0.
• All DCS Voice threads to use all other cores except Core 0.
• All supposable DCS Disk I/O "ucrtbase" and "ntdll" threads to use all other cores except Core 0 and Core 1.

 

You could additionally use Process Lasso to set the Process Affinity of all Processes to avoid using Core 0, but for the effect it would have I decided not to combine that program into the mix for this demo because it could interfere in another way that I didn't want to lose time on, that may be needed if you have something that's taking up CPU that you can't get rid of while doing this test.

 

Once you set the affinities of all the threads, it is unfortunately a tedious one-by-one task at least for now, then disable the automatic refresh Process Hacker (Main Windows -> View -> Refresh Automatically). It takes DPC latency which may create sound distortion and popping could affect the test perhaps. You can manually refresh with F5 when necessary, but the main windows has to be in-focus.

 

You should reboot the PC between tests with different Thead affinity, or simulate it by stopping DCS, clearing up Windows Standby Memory (Cached Memory*) and then starting up DCS again.

You can clear up Standby Memory by using various utilities such as RAMMap, but others fall from my memory right now, or more conveniently use a tiny purpose built CMD utility which you can find source code on the web, however that's I suppose impractical for most of you here as you would need to compile it yourself.

 

I build this one myself quite some time ago and been using it for various purposes such as this:

Attachement: https://forums.eagle.ru/attachment.php?attachmentid=230970&stc=1&d=1585447232

 

To verify the security of this utility read and follow the following:

StandbyCacheFlusher.exe: https://www.virustotal.com/gui/file/1d1c863669bff562a6e757fba7d28e51fac7bdd05fb5f1db180f186f8a1b0ab3/detection

The zipped attachement: https://www.virustotal.com/gui/file/926976eaa1c90a0f6894ca9f47cca2cf30e6efd8f0615d86121cd8c5f1485327/detection

Example hash checker: https://www.nirsoft.net/utils/hash_my_files.html

 

When you download the attachement check that the file hash is: 926976eaa1c90a0f6894ca9f47cca2cf30e6efd8f0615d86121cd8c5f1485327

Once you unpack the exe check that it's hash is: 1d1c863669bff562a6e757fba7d28e51fac7bdd05fb5f1db180f186f8a1b0ab3

If your antivirus is working correctly it should in theory either catch it as you open the zip, or once you extract the exe, prior to you executing it.

 

You can also take the time to upload it yourself to virustotal to have it rechecked and that way you can check it and the hash all in one go in the details page.

Dates of exe should be:

Creation Time 2017-06-06 10:56:44

First Submission 2020-03-27 11:28:51

 

 

You could check that the flushing has worked to be sure, you can do that in the standard windows Resource Monitor or Task Manager:

 

 

* Note: Cached Memory includes Standby Memory. Cached Memory counter also includes Modified Memory that's why it'll show a bit higher but that's normal, but the numbers of both of these should be pretty low versus as per usual after you clear it and then it should be ready to start up DCS again.

 

If you are still unsure about using this tiny utility method then just use the other option such as rebooting the PC.

 

 

 

--------------------------------------------------------------------------------------------------------

///////////////////////////////////////////////////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------------------------------

 

Options and PC details:

 

Spoiler

 

My system specs:

 

 

DCS installed on SSD: Samsung 860 EVO

 

------------------
System Information
------------------
     Time of this report: 3/27/2020, 11:05:14
            Machine name: REDACTED
              Machine Id: REDACTED
        Operating System: Windows 10 Enterprise 2016 LTSB 64-bit (10.0, Build 14393) (14393.rs1_release.160715-1616)
                Language: Slovenian (Regional Setting: Slovenian)
     System Manufacturer: System manufacturer
            System Model: System Product Name
                    BIOS: BIOS Date: 05/06/14 19:08:43 Ver: 47.01
               Processor: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz (4 CPUs), ~3.6GHz
                  Memory: 32768MB RAM
     Available OS Memory: 32710MB RAM
               Page File: 12050MB used, 28851MB available
             Windows Dir: C:\Windows
         DirectX Version: DirectX 12
     DX Setup Parameters: Not found
        User DPI Setting: Using System DPI
      System DPI Setting: 96 DPI (100 percent)
         DWM DPI Scaling: Disabled
                Miracast: Available, with HDCP
Microsoft Graphics Hybrid: Not Supported
          DxDiag Version: 10.00.14393.0000 64bit Unicode

---------------
Display Devices
---------------
         Card name: Radeon (TM) RX 480 Graphics
      Manufacturer: Advanced Micro Devices, Inc.
         Chip type: AMD Radeon Graphics Processor (0x67DF)
          DAC type: Internal DAC(400MHz)
       Device Type: Full Device
        Device Key: Enum\PCI\VEN_1002&DEV_67DF&SUBSYS_E347174B&REV_C7
     Device Status: 0180200A [DN_DRIVER_LOADED|DN_STARTED|DN_DISABLEABLE|DN_NT_ENUMERATOR|DN_NT_DRIVER] 
Device Problem Code: No Problem
Driver Problem Code: Unknown
    Display Memory: 24527 MB
  Dedicated Memory: 8172 MB
     Shared Memory: 16354 MB
      Current Mode: 2560 x 1440 (32 bit) (75Hz)
      Monitor Name: Generic PnP Monitor
     Monitor Model: PB328
        Monitor Id: ACI32A5
       Native Mode: 2560 x 1440(p) (59.951Hz)
       Output Type: Displayport External
       Driver Name: C:\Windows\System32\DriverStore\FileRepository\c0351300.inf_amd64_c0a7387fe716cd77\B351053\aticfx64.dll,C:\Windows\System32\DriverStore\FileRepository\c0351300.inf_amd64_c0a7387fe716cd77\B351053\aticfx64.dll,C:\Windows\System32\DriverStore\FileRepository\c0351300.inf_amd64_c0a7387fe716cd77\B351053\aticfx64.dll,C:\Windows\System32\DriverStore\FileRepository\c0351300.inf_amd64_c0a7387fe716cd77\B351053\amdxc64.dll
Driver File Version: 26.20.15015.1007 (English)
    Driver Version: 26.20.15015.1007
       DDI Version: 12
    Feature Levels: 12_0,11_1,11_0,10_1,10_0,9_3,9_2,9_1
      Driver Model: WDDM 2.1
Graphics Preemption: Primitive
Compute Preemption: DMA
          Miracast: Not Supported
Hybrid Graphics GPU: Not Supported
    Power P-states: Not Supported
 Driver Attributes: Final Retail
  Driver Date/Size: 17. 01. 2020 01:00:00, 1947704 bytes
       WHQL Logo'd: n/a
   WHQL Date Stamp: n/a
 Device Identifier: {D7B71EE2-249F-11CF-1477-E0F97CC2D735}
         Vendor ID: 0x1002
         Device ID: 0x67DF
         SubSys ID: 0xE347174B
       Revision ID: 0x00C7
Driver Strong Name: oem0.inf:cb0ae414b19b97d1:ati2mtag_Polaris10DS:26.20.15015.1007:pci\ven_1002&dev_67df&rev_c7
    Rank Of Driver: 00D12000
       Video Accel: Unknown
       DXVA2 Modes: DXVA2_ModeMPEG2_VLD  DXVA2_ModeMPEG2_IDCT  DXVA2_ModeH264_VLD_NoFGT  DXVA2_ModeHEVC_VLD_Main  DXVA2_ModeH264_VLD_Stereo_Progressive_NoFGT  DXVA2_ModeH264_VLD_Stereo_NoFGT  DXVA2_ModeVC1_VLD  DXVA2_ModeMPEG4pt2_VLD_AdvSimple_NoGMC  DXVA2_ModeHEVC_VLD_Main10  
     Deinterlace Caps: n/a
      D3D9 Overlay: Not Supported
           DXVA-HD: Not Supported
      DDraw Status: Enabled
        D3D Status: Enabled
        AGP Status: Enabled
          MPO Caps: Not Supported
       MPO Stretch: Not Supported
   MPO Media Hints: Not Supported
       MPO Formats: Not Supported

-------------
Sound Devices
-------------
           Description: Speakers (2- Realtek High Definition Audio)
Default Sound Playback: Yes
Default Voice Playback: Yes
           Hardware ID: HDAUDIO\FUNC_01&VEN_10EC&DEV_0892&SUBSYS_104384FB&REV_1003
       Manufacturer ID: 1
            Product ID: 100
                  Type: WDM
           Driver Name: RTKVHD64.sys
        Driver Version: 6.00.0001.8186 (English)
     Driver Attributes: Final Retail
           WHQL Logo'd: n/a
         Date and Size: 14. 06. 2017 00:00:00, 5826560 bytes
           Other Files: 
       Driver Provider: Realtek Semiconductor Corp.
        HW Accel Level: Basic
             Cap Flags: 0x0
   Min/Max Sample Rate: 0, 0
Static/Strm HW Mix Bufs: 0, 0
Static/Strm HW 3D Bufs: 0, 0
             HW Memory: 0
      Voice Management: No
EAX(tm) 2.0 Listen/Src: No, No
  I3DL2(tm) Listen/Src: No, No
Sensaura(tm) ZoomFX(tm): No

 

 

 

Secondary test system specs:

 

DCS installed on HDD - WD Red 3TB (MFD: 2018)

 

------------------
System Information
------------------
Time of this report: 3/26/2020, 21:22:38
      Machine name: REDACTED
  Operating System: Windows 7 Ultimate 64-bit (6.1, Build 7601) Service Pack 1 (7601.win7sp1_gdr.150928-1507)
          Language: Slovenian (Regional Setting: Slovenian)
System Manufacturer: Gigabyte Technology Co., Ltd.
      System Model: Z68XP-UD4
              BIOS: Award Modular BIOS v6.00PG
         Processor: Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz (4 CPUs), ~3.3GHz
            Memory: 16384MB RAM
Available OS Memory: 16368MB RAM
         Page File: 3640MB used, 29108MB available
       Windows Dir: C:\Windows
   DirectX Version: DirectX 11
DX Setup Parameters: Not found
  User DPI Setting: Using System DPI
System DPI Setting: 96 DPI (100 percent)
   DWM DPI Scaling: Disabled
    DxDiag Version: 6.01.7601.17514 32bit Unicode

---------------
Display Devices
---------------
         Card name: NVIDIA GeForce GTX 1070
      Manufacturer: NVIDIA
         Chip type: GeForce GTX 1070
          DAC type: Integrated RAMDAC
        Device Key: Enum\PCI\VEN_10DE&DEV_1B81&SUBSYS_143519DA&REV_A1
    Display Memory: 4095 MB
  Dedicated Memory: 3072 MB
     Shared Memory: 1023 MB
      Current Mode: 1920 x 1080 (32 bit) (144Hz)
      Monitor Name: Generic PnP Monitor
     Monitor Model: ASUS VG278HE
        Monitor Id: ACI27E3
       Native Mode: 1920 x 1080(p) (60.000Hz)
       Output Type: DVI
       Driver Name: nvd3dumx.dll,nvwgf2umx.dll,nvwgf2umx.dll,nvd3dum,nvwgf2um,nvwgf2um
Driver File Version: 26.21.0014.3039 (English)
    Driver Version: 26.21.14.3039
       DDI Version: 11
      Driver Model: WDDM 1.1
 Driver Attributes: Final Retail
  Driver Date/Size: 4/18/2019 14:57:06, 21646816 bytes
       WHQL Logo'd: n/a
   WHQL Date Stamp: n/a
 Device Identifier: {D7B71E3E-58C1-11CF-B571-3B341BC2D735}
         Vendor ID: 0x10DE
         Device ID: 0x1B81
         SubSys ID: 0x143519DA
       Revision ID: 0x00A1
Driver Strong Name: oem7.inf:NVIDIA_Devices.NTamd64.6.1:Section110:26.21.14.3039:pci\ven_10de&dev_1b81
    Rank Of Driver: 00E62001
       Video Accel: ]removed lots of unimportant guid gibberish
      D3D9 Overlay: Supported
           DXVA-HD: Supported
      DDraw Status: Enabled
        D3D Status: Enabled
        AGP Status: Enabled

-------------
Sound Devices
-------------
           Description: Speakers (ASUS Xonar D1 Audio Device)
Default Sound Playback: Yes
Default Voice Playback: Yes
           Hardware ID: PCI\VEN_13F6&DEV_8788&SUBSYS_834F1043&REV_00
       Manufacturer ID: 1
            Product ID: 100
                  Type: WDM
           Driver Name: cmudaxp.sys
        Driver Version: 0.00.0008.1823 (English)
     Driver Attributes: Final Retail
           WHQL Logo'd: n/a
         Date and Size: 6/2/2015 12:23:06, 2735616 bytes
           Other Files: 
       Driver Provider: ASUSTeK
        HW Accel Level: Basic
             Cap Flags: 0x0
   Min/Max Sample Rate: 0, 0
Static/Strm HW Mix Bufs: 0, 0
Static/Strm HW 3D Bufs: 0, 0
             HW Memory: 0
      Voice Management: No
EAX(tm) 2.0 Listen/Src: No, No
  I3DL2(tm) Listen/Src: No, No
Sensaura(tm) ZoomFX(tm): No

 

 

That Nvidia card is a better one, while the CPU is weaker, and that GPU does eat through the Black Engine Exhaust Smoke a bit better but still that problems is present there nontheless, by the way. (there's a separate thread on Black Smoke)

 

However, I did not test anything beside 2.5.6 OpenBeta yet, I plan to do the 2.5.5 release and earlier if possible but I just wanted to get the initial findings out of the way. I am as I'm able now for some time also use the secondary PC to do checks to reinforce these reports.

 

 

The default affinity vs workaround one doesn't have that much of an impact that I could quickly see when switching airfield views with F11, but I might doulbe check that again, making a video by using affinity to delegate OBS to only 2 cores (for example the 2 cores the DCS Disk I/O stuff was put on) would produce a heavily tained recording which would not reflect reality under the circumstances this workaround is meant to be used. What happens? Well, it takes 10 seconds for the assets to pop in which is kinda crazy, even tho the sim doesn't lag, the camera can be rotated freely.

 

I would really need like a monitor capture hardware for this kind of stuff.

 

 

--------------------------------------------------------------------------------------------------------

///////////////////////////////////////////////////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------------------------------

 

Update1: Feb 2021 Major Reword

ProcessHacker Nightly Download

StandbyCacheFlusher.zip

TRACK_DCS-256OB_TEST_CAUC_A10C_BareBone-TGP-MAV-TerrainHorizon-DiskIO-Lag_27March2020.trk

 

 

--------------------------------------------------------------------------------------------------

/////////////////////////////////////////////////////////////////////////////////////////

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

--------------------------------------------------------------------------------------------------

 

 

WIP

 

Edited February 10, 2021 by Worrazen

[Worrazen]

Reserved

[Worrazen]

Nevada Case Examples: (in progress, planning to also display CPU0 graph where only the main thread resides in "workaround affinity" cases)

 

 

 

Latest notes:

 

It may be more things at play, don't even need to be in cockpit and view the TGP to make some of the stutters happen. But this makes sense that the asset loading works all around even if you're not looking at it directly, so that's probably not unexpected then.

 

Not all Disk usage spikes cause a FPS drop, right now I have no idea, it's pretty random and unexpected, it may be that some specific data causes it under specific circumstances, perhaps the initial rendering of it is costly and takes a chunk, it goes into more speculation waters from here on out, but I'm going to keep an eye on this ongoing, but without rushing to any results or conclusions.

 

 

Quick Update

 

Obviously it's about a specific component on the main thread, because you can have the thread 100% but for a different reason most likely graphics api and drivers, it'll just limit the max FPS most likely, but it won't stutter or interrupt anything, usually the main thread does a lot more as FPS increases.

 

The reason gui locks during loading screen is probably understandable but I think with a different approach this and other such cases could be avoided in future, it's not that big of a deal tho, and now when I'm using workaround affinity, i've seen it work better and not cause any errors, it's kidna random so I can't call it.

 

 

 

I'll look more at the terrain stutters deeper later.

 

 

 

------------------Update:

Discovered something new with one of the special ucrtbase threads, it's a finding on it's own, but I'll post it in the same topic, quite interesting.

Edited March 31, 2020 by Worrazen

[Elnocho3]

I’m sorry, but I completely stopped using other sims because I was doing this level of tweaking. Spending more time reading, tweaking software and hardware, than actual flying.

 

So I appreciate your efforts, but is there a TLDR?

[BIGNEWY]

Great post Worrazen

 

The team have seen it and will look into it as part of our performance review.

 

thank you

[sniperwolfpk5]

Well done Worrazen. ED should hire you for QA and performance testing.

[HC_Official]

Well done Worrazen. ED should hire you for QA and performance testing.

 

 

I agree a great bit of analysis from Worrazen

[Worrazen]

There's another separate thing I mentioned I wanted to report, but couldn't do it in time, I just have to do some other chores around the house which again includes disconnecting the PC for a few days or more.

 

That one may be even bigger than this, but I have to give it more time so that I don't keep rewriting the posts and correcting myself, one of my bad habits :music_whistling:.

 

I'm not sure anymore if it'll be worthwhile posting that in the same thread, while it's about the File/Disk I/O and threading, it doesn't have relation directly on the outside ... infact when I come back I'll separate the MSI Overlays guide into a more general one, there's a few paragraphs I see I misplaced and the formatting isn't the best

.

 

I’m sorry, but I completely stopped using other sims because I was doing this level of tweaking. Spending more time reading, tweaking software and hardware, than actual flying.

 

So I appreciate your efforts, but is there a TLDR?

 

Sorry, not this time around, I did a general "report" on threading a weeks ago because I've seen a lot of external communities bringing up the whole single-threaded idea and while that is somewhat true it's really a different story these days and there's about 3 really busy threads as well as a bnch of Disk I/O & Memory threads so a Quad-Core CPU should be well worth the dime and it's not that bad really, it infact exceeded my expectations.

 

Because this is primairly a "bug report" and focusing on technical details that are neceessary to have right, from what is possible from my side at least, it is not meant for everyone. The reason why it seems it's a lot is because I wrote it as a guide for fellow players who are browsing this subforum that are kinda interested but may not be willing to get too deep into or don't know where to start, it's can be a grueling job and it's not that interesting such as modding, but I thought perhaps it's not interesting because of less guides and ease of getting into it, so why not also provide some explanation from which it can help you understand this threading and the basic CPU perf counters for good.

 

Hopefully reporting performance bugs comes with more details in the future.

 

If one does not have interest in replicating this track/guide the only TLDR can be the description, or a long title, which I agree I should have posted as the first sentence; the DCS Disk I/O & Mem/Data Management threads* seem to be competing for CPU Time and because throttling them and allowing the main thread to have a full core all to it self seems to be getting rid of the stutters is what lead me to call the title like I did, but these threads could be interrupting something else as well, this is usually always an educated guess more than some kind of proof and definitive answer.

 

The "Mem/Data Management" part is my latest speculation because they also seem to be active anytime there's any shuffling of most-needed textures when you're scrolling/zooming in various views or the F10 map -->> even when there are no Disk reads(activity) so it must be in my opinion right now, also managing the assets/textures/terrain objects around in RAM/VRAM, I'll show this later or in an newer thread part of where it is most relevant.

[BitMaster]

Would you think that a high core count machine and the fastest I/O path available would show a lesser performance impact ? Thinking of AMD x570 chipset and Gen4 NVMe along with a fast 3000 series CPU with more than 4 cores ( they all have 6 or more ), say a 12 or 16 core monster ?

 

Vice versa, would a 4-core Sata SSD based system suffer more than the above ?

 

Throughput is everything, hammer the bits down the pipe like there is no other day ;)

[Worrazen]

ˆˆ From all the ucrtbase threads, only like 5 or so are usually active when traversing terrain or moving the F10 maps, is what I saw later on but I wrapped up shortly after. I didn't actually plan to do this but I just had to stay with DCS to get the most out of the trials and getting some bug reports in.

 

Maybe with a 12 or more core CPU this may just work and a workaround wouldn't be needed, but what if the game spawns even more I/O threads haha, DCS simply needs to detect amount of cores properly, whether HT/SMT is enabled and adjust amount of I/O threads if that's possible, however it is designed, perhaps not all threads produce the same type of work.

 

Whatever the threads are, streaming textures should never interrupt execution, that is the whole point of streaming, that you can continue while things gradually come in

 

It's clearly not suttering because of draw-calls and driver and API overhead if that were the case the sutters would be everywhere all the time, but it's like more specific and random-seeming sometimes (that's because the loaded radius area moves with the camera 360 degrees, don't need to look at something directly to trigger things behind scenes)

 

Sent from my phone SM-G900F

[The_Nephilim]

Well I tried this method with Process Lasso I had put all processes on cores 2-3, I put DCS on cores 0-1. I tried running Through the inferno and flew the f18 off the carrier..

 

I then tried the same with project lasso disabled it was more smooth without Process Lasso.. I did get more FPS when running PL but it was not as smooth looking or feeling..

 

I would like to point out I tried this in VR so maybe something there makes it not run as smooth like stated in OP IDK..

 

Either way it led to some intresting points from the Project I am working on so this was helpful..

 

I was wondering I if I used PL again and put all processes on 1 core and DCS on 3 how would that work.. I may try but later for know it was a bust for me in VR YMMV. :)

 

 

EDIT: well I did try putting the game Only on 3 Cores and the rest on 1 core, it worked pretty good in my initial test.. I think in this case more cores would be very helpfull.. I think maybe 10 cores should be sufficient for what I want to do.. wierd it did not work well when the game was only on 1-2 cores..

Edited April 3, 2020 by The_Nephilim

[Worrazen]

^^ Process Lasso does not support per-thread CPU affinity selection the last time I checked, which was a week ago.

 

To my knowledge so far only Process Hacker can do it. I literally made this thread the "moment" I found about it so I didn't spend any time looking for any other utility, (as you may or may not know I've been posting about performance here for quite some time but without any such experience prior so DCS was my learning test-bed for all of this, which means some old posts are probably wrong and obsolete, I kept going and correcting things as time passed. I've eventually dug in deeply to figure out quite a bit and then I figured in order to attempt certain tests one must be able to have per-thread CPU affinity control, which I kept "searching" for many months, until now, and this is my first report with the help of that feature.)

 

You can't attempt this trick with only per-process CPU affinity, it only emulated how many CPUs the whole process and all of it's threada can use, kinda as if you installed a diffrent CPU.

 

You were seeing correct results, 3 cores are indeed better than 2.

 

This report was still rushy a bit, I'll organize myself next time and I'm probably going to get another SSD so I can have both Release and OpenBeta installed at the same time and other things, I have already figured I need a good template in photoshop in which key details to explain the context of a screenshot are pre-entered and then I can just paste in the appropriate ones, because this can get time consuming and complicated once there's a lot of test cases to compare with, keeping things sync in one's mind can take effort heh, if it's not well documented

 

I also started thinking about a hardware caputure device, but I don't expect price to be easy ... because there's no video to see the difference here, I can't make it with SW because it would modify the behavior and render the video invalid, taking screenshots is much better but still leaves those drop-spikes which can easily get confused with if one forgets about it just for a moment.

 

Sent from my phone SM-G900F

Edited April 3, 2020 by Worrazen

[The_Nephilim]

^^ Process Lasso does not support per-thread CPU affinity selection the last time I checked, which was a week ago.

 

To my knowledge so far only Process Hacker can do it. I literally made this thread the "moment" I found about it so I didn't spend any time looking for any other utility, (as you may or may not know I've been posting about performance here for quite some time but without any such experience prior so DCS was my learning test-bed for all of this, which means some old posts are probably wrong and obsolete, I kept going and correcting things as time passed. I've eventually dug in deeply to figure out quite a bit and then I figured in order to attempt certain tests one must be able to have per-thread CPU affinity control, which I kept "searching" for many months, until now, and this is my first report with the help of that feature.)

 

You can't attempt this trick with only per-process CPU affinity, it only emulated how many CPUs the whole process and all of it's threada can use, kinda as if you installed a diffrent CPU.

 

You were seeing correct results, 3 cores are indeed better than 2.

 

This report was still rushy a bit, I'll organize myself next time and I'm probably going to get another SSD so I can have both Release and OpenBeta installed at the same time and other things, I have already figured I need a good template in photoshop in which key details to explain the context of a screenshot are pre-entered and then I can just paste in the appropriate ones, because this can get time consuming and complicated once there's a lot of test cases to compare with, keeping things sync in one's mind can take effort heh, if it's not well documented

 

I also started thinking about a hardware caputure device, but I don't expect price to be easy ... because there's no video to see the difference here, I can't make it with SW because it would modify the behavior and render the video invalid, taking screenshots is much better but still leaves those drop-spikes which can easily get confused with if one forgets about it just for a moment.

 

Sent from my phone SM-G900F

 

 

Well when I first read the thread I was mistaken as to what exactly you where doing.. I reread and seen why you used process hacker.. I did try what you said and what am I supposed to get from doing that which was very tedious hehe!!

 

it seemed ok but still got some stutters inland a bit over Causcus.. Maybe if I combined what I did with Process Lasso to put all other processes on another CPU Besides 0 it would have helped with the stutter..

 

I did see as post earlier today where the user had a new CPU nd a 2080S and it still had issues in 2.5.6.. Hopefully they can optimise whast is wrong with 2.5.6 and it will run like it did in 2.5.5..

[PetRock]

I just wanted to say thanks Worrazen - This problem was causing some major FPS lock-ups when I went to 2.5.6 for the first time which was compounded by my c: drive being compressed when I upgraded the OS a month ago and needed just a tiny bit more room for the swap.

 

 

I watched the system resource monitor while sitting in the hangar on the ground to ID if there was a GPU or CPU bottleneck but the drop/freezes matched 1:1 with 100% load spike to C: .

 

 

Now that I have uncompressed C:, FPS is much better, but I still get drops to 20-30 FPS that now coincide with spikes to my SSD game drive (F: ) and now almost zero usage on C:

 

 

Hope this tidbit helps ED and you continue your investigation. At least its playable now.

Edited April 4, 2020 by PetRock
formating

[SPAS79]

1) I'm being massively whoooooshed here, and I tend to be pretty geeky with this kind of stuff, but this is too much geekiness for me.

 

2) [RANT] as said by a previous poster, I have left other sims because I was sick and tired of workarounds, tweaks, third-party addons to fix this and that.

 

BUT that was because THAT sim was abandoned, the development stopped, the devs laid off (we all know the story). DCS is in active development, so I would expect better, although I feel DCS is slowly evolving into this user-level "thereIfixedIt" model, and while your analysis is awesome I would rather migrate to another sim (if and when Microprose decides to raise back from the dead, or the unspeakable BMS thing) rather than spending hours on end trying to fix something broken.

 

Hell, DCS 2.5.5 was working so smoothly and beautifully it was unbelievable. I do appreciate 2.5.6 is OB and everything, but I feel like the cost/benefit for the new lighting system is a bit uneven... I've lost about 30% performance for some better lighting (which was OK before, anyway).

 

I don't like where DCS is going and the fact that devs are blaming it on leftover spaghetti from people who left the company. If it is your product and your company, you should fix it and not expect the users to go through hoops to make your product work for them.

 

All in all, good analysis and I hope ED will take notes (as I'm sure they'll do) and fix things (here not so sure, but this is just my opinion).

I will wait patiently and keep tinkering with my external instruments exports and Helios, which is my current favorite timesink.

Edited April 7, 2020 by SPAS79

[sniperwolfpk5]

1) I'm being massively whoooooshed here, and I tend to be pretty geeky with this kind of stuff, but this is too much geekiness for me.

 

2) [RANT] as said by a previous poster, I have left other sims because I was sick and tired of workarounds, tweaks, third-party addons to fix this and that.

 

BUT that was because THAT sim was abandoned, the development stopped, the devs laid off (we all know the story). DCS is in active development, so I would expect better, although I feel DCS is slowly evolving into this user-level "thereIfixedIt" model, and while your analysis is awesome I would rather migrate to another sim (if and when Microprose decides to raise back from the dead, or the unspeakable BMS thing) rather than spending hours on end trying to fix something broken.

 

Hell, DCS 2.5.5 was working so smoothly and beautifully it was unbelievable. I do appreciate 2.5.6 is OB and everything, but I feel like the cost/benefit for the new lighting system is a bit uneven... I've lost about 30% performance for some better lighting (which was OK before, anyway).

 

I don't like where DCS is going and the fact that devs are blaming it on leftover spaghetti from people who left the company. If it is your product and your company, you should fix it and not expect the users to go through hoops to make your product work for them.

 

All in all, good analysis and I hope ED will take notes (as I'm sure they'll do) and fix things (here not so sure, but this is just my opinion).

I will wait patiently and keep tinkering with my external instruments exports and Helios, which is my current favorite timesink.

 

Well said

[Liam98]

I appreciate the effort but holy f*ck nobody is trying to read all that. Most DCS players arent interested in 99% of what you've written. Please include a TLDR; not really interested in anything else

[Taz1004]

21 hours ago, Liam98 said:

I appreciate the effort but holy f*ck nobody is trying to read all that. Most DCS players arent interested in 99% of what you've written. Please include a TLDR; not really interested in anything else

 

You might be surprised that many DCS players are used to reading.  Such as flight manual and detailed mission briefing.

Edited January 5, 2021 by Taz1004

[mwlue]

4 hours ago, Taz1004 said:

 

... many DCS players are used to reading ...

 

 

Yep, for those tired of reading.... we are so sorry,  should probably go for ace combat in console.  It had fewer buttons & switches to take care of, almost no readings required at all.

Tons of fun as well.

[Worrazen]

On 1/4/2021 at 7:25 AM, Liam98 said:

I appreciate the effort but holy f*ck nobody is trying to read all that. Most DCS players arent interested in 99% of what you've written. Please include a TLDR; not really interested in anything else

 

Yes, rest assured I'm well aware of that that it's mostly computer technical and not really DCS-content related, tho most people here should be well used to reading long pieces of material as you heard. It was meant for the beta-tester and the crowds that are regularly reading this subforum, but more importantly for the devs themselfs to get as much detail and overal picture that could help them out getting to the solution faster. 

Plus, sometimes I like doing this kind of stuff for my own kicks when time allows (unless if I'd be doing this as business work then I wouldn't had as much time and energy writing it in such a style or at all)

 

I'm pretty close getting my systems up and running again, tho things keep delaying as new things pop-up, I was just updating my router's firmware and doing a whole re-configuration with custom plugins and stuff, hopefully I can finish setting up in the freshly installed Win10 and get back to DCS, so I'll have a new report I was not able to finish from half a year ago coming up soon, that is if it wasn't fixed already, I didn't had DCS installed during all of this.

 

 

Edited January 6, 2021 by Worrazen

[Worrazen]

On the other hand I'm preparing another report in this area, but since I'm close I'll update this thread as well because I indeed didn't word things bestly here, first I didn't really want to include a rough MSI Afterburner guide into this post, and secondly I went on a tangent about the not too important side effect of "unresponsive program" in DCS loading screens, that has since resolved it self, by either DCS it self or a combination of moving on to a more later OS version.

 

I'll take perhaps another shot at re-evaluating the core issue in this thread, whether things have improved, whether the claims still hold true and ofcourse to fix my own mistakes if there's any and make the OP more clearer.

 

LOADING SCREEN SIDE-EFFECT UPDATE FEB 2021:

New fresh Windows 10 Professional 2004 / 20H1 (PC2 Specs in signature below) + DCS 2.5.6.60966 -> I do not get the Windows warning dialog for "unresponsive program" during loading times anymore, it is now working correctly. The mouse cursor stays normal, no busy-loading cursor animation, the window stays normal fullscreen and loading continunes as normal without windowing issues. Great!

Edited February 11, 2021 by Worrazen