Jump to content

Sporadic and prolonged unexplained FPS drops in VR (with GPU/perf capture)


Recommended Posts

There have been many reports of unexplained prolonged FPS drops in VR. I have provided what I hope is a consistent repro along with a GPU/perf capture later on below, but common symptoms of this issue seem to include:

  1. Happens randomly, but common triggers seem to be actions that cause temporary performance blips (that then turn into sustained FPS drops)
  2. FPS goes from something stable and normal (e.g. 30/45fps) to significantly lower and unplayable (e.g. 15-25fps)
  3. FPS drop lasts for many minutes, sometimes requiring exiting the game to recover
  4. VR-specific
  5. Alt-tabbing away from and back to DCS sometimes resolves the issue
  6. Returning to the main menu still exhibits abnormally low FPS

I think those last three points are important, as they point to a software bug, not a hardware bottleneck. FPS would return to normal at the main menu, and alt-tabbing wouldn't fix the issue, if this were simply a problem with users' machines being incapable of sustaining adequate framerates. Furthermore, the issue seems to be VR-specific, pointing to a VR-specific bug as opposed to a hardware capacity issue.

Here's a collection of forum posts that I believe exhibit this issue (there are more):

 

The most consistent repro that I can find is to:

  1. Load into main menu
  2. Observe 90fps
  3. Load into Cold Start instant action on Marianas map in F/A-18C sim (this is a good starting point, since you're on a carrier, and can see the island in the distance)
  4. Observe stable 45fps
  5. Slowly turn your head to the left, until you're looking behind your left shoulder
  6. Slowly turn your head to face back forward, then right, until you're looking behind your right shoulder this time
  7. Turn your head back to face forward
  8. Observe ~14 fps
  9. Quit to main menu
  10. Observe ~25 fps

This is on current stable Steam DCS (also on beta), Windows 10, a stock RTX 3080 (driver 512.95, doesn't seem to matter), stock Ryzen 5950x, HP Reverb G2, WMR 2000.21051.1282.0 (version doesn't seem to matter), OpenXR 110.2204.7007 (also reproduces on SteamVR), 100% render scale (3168x3096 pixels per eye), motion reprojection always on, and the following in-game settings:

2022-06-02 08:30:19.529 INFO    APP: options.graphics =
{
	['messagesFontScale'] = 1;
	['forestDetailsFactor'] = 0.75;
	['rainDroplets'] = true;
	['LensEffects'] = 0;
	['box_mouse_cursor'] = true;
	['anisotropy'] = 4;
	['water'] = 1;
	['motionBlur'] = 0;
	['outputGamma'] = 2;
	['aspect'] = 1.6666666666667;
	['lights'] = 2;
	['shadows'] = 4;
	['MSAA'] = 1;
	['SSAA'] = 0;
	['civTraffic'] = '';
	['clutterMaxDistance'] = 0;
	['cockpitGI'] = 0;
	['terrainTextures'] = 'max';
	['multiMonitorSetup'] = '1camera';
	['shadowTree'] = false;
	['chimneySmokeDensity'] = 0;
	['fullScreen'] = false;
	['forestDistanceFactor'] = 0.75;
	['useDeferredShading'] = 1;
	['DOF'] = 0;
	['clouds'] = 1;
	['sceneryDetailsFactor'] = 1;
	['width'] = 1280;
	['flatTerrainShadows'] = 0;
	['visibRange'] = 'Ultra';
	['textures'] = 2;
	['SSLR'] = 0;
	['effects'] = 3;
	['SSAO'] = 0;
	['secondaryShadows'] = 0;
	['sync'] = false;
	['scaleGui'] = 1;
	['preloadRadius'] = 150000;
	['height'] = 768;
	['heatBlr'] = 0;
	['terrainMapMFD'] = 
	{
		['distance'] = 
		{
			['mapLodDistance3'] = 200000;
			['mapLodDistance2'] = 100000;
			['mapLodDistance0'] = 25000;
			['mapLodDistance1'] = 50000;
		};
	};
};

 

This behavior doesn't really make sense. I'm sure that the act of looking right on the Marianas map is exceeding the VRAM on my 3080, causing a temporary blip as stuff has to be paged out of VRAM for it, but when I look back forward, everything should get paged back in automatically (at least on DX11) and return to normal. And indeed, in the captures below, there isn't any persistent VRAM paging going on that would explain the severe drop in FPS. Similarly, when returning to the main menu, instead of getting 90fps like I was before, I'm at 25fps, and nothing strenuous is going on (I haven't grabbed a capture of this yet as it's much more difficult to get a capture of due to needing to alt-tab to start the capture, and alt-tab typically clears the issue in the main menu);

I took a GPUView capture of the issue on the Marianas map (happy to provide to private support), and it clearly shows some kind of problem with how DCS is scheduling it's main thread. In summary, it appears to be sleeping too long and doing nothing. This isn't a problem that requires advanced multithreading, DX12, better VRAM management, etc... It simply appears to be an issue with DCS waiting too long to start rendering the next time.

 

Here is a capture of a normal, stable 30 fps for me after loading into Marianas (higher settings than I mentioned previously, but still a decent 30fps):

rqNTRoB.png

What I've gathered from this is that:

  1. In VR, DCS has a thread started from WinXrRuntime.dll that runs at 90Hz (or whatever the refresh rate of your headset is)
    1. You can see the row for this in the bottom left (labeled WinXrRuntime.dll)
    2. The blue boxes are when this thread is running on a CPU core. You can see a total of 14 of them, because this capture window covers about 160 milliseconds, so at 90hz we'd expect about 14 occurrences in a 160 millisecond time period
    3. I believe this is due to xrWaitFrame (https://www.khronos.org/registry/OpenXR/specs/1.0/man/html/xrWaitFrame.html), which must be called by the application to synchronize frame submissions with my headset. So this thread is probably calling xrWaitFrame on a loop to do that, which is why it's running at 90Hz
  2. The DCS.exe thread is running at 30Hz, every 3rd time that the WinXrRuntime.dll thread is running
    1. The WinXrRuntime.dll thread is seemed to be used to (among other things) schedule the main DCS.exe thread to run (i.e. this is how DCS ensures it synchronizes its frame submissions with your VR headset)
    2. You can see the row for this a bit above WinXrRuntime.dll, labeled as DCS.exe
    3. Similarly, the blue boxes are when this thread is running on a CPU core, which aligns with every 3rd time that WinXrRuntime.dll runs
  3. In this capture, when the DCS.exe thread is signaled to run, it executes for about 16ms
    1. This isn't fast enough to run at 90Hz
    2. As noted above, it's running at every 3rd time that the WinXrRuntime.dll thread is running
    3. I.e. this is running at 30Hz, or 30fps. Which with motion reprojection, is perfectly stable and playable
    4. This could run at 45Hz, however my GPU wouldn't be able to keep up, so it would be a bottleneck and my FPS wouldn't improve. You can tell this from the Device Context and RTX 3080/Hardware Queue 3D rows, which show that the GPU is being kept busy ~85% of the time. The GPU work scheduled to the Device Context queue also takes about 28ms to drain, so it's limited to about 30fps, and there's no point to the DCS.exe thread running more often (at least, not for scheduling GPU work)
  4. In summary, the main DCS.exe thread is running at the ideal 30Hz for how my hardware is performing, and I'm getting a reasonably smooth 30fps with motion reprojection

 

Now, let's compare this with a capture of an abnormal, unstable ~22fps for me, on the same Marianas map, after briefly turning to the right and then looking forward in the same exact direction (i.e. should be rendering the same stuff as earlier in the 30fps capture):

n9oQH0t.png

There are a few notable differences from the earlier capture that was stable at 30fps:

  1. The WinXrRuntime.dll thread is still running at the expected 90Hz
  2. The DCS.exe thread is running at 22.5Hz, every 4th time that the WinXrRuntime.dll thread is running (instead of every 3rd)
  3. In this capture, when the DCS.exe thread is signaled to run, it executes for about 23ms
    1. This is higher than in the good capture (23ms vs. 16ms)
    2. As noted above, it's running at every 4th time that the WinXrRuntime.dll thread is running
    3. I.e. this is running at 22.5Hz, or 22.5fps. Even with motion reprojection, this isn't very playable (it also varies a lot and isn't stable)
    4. This could run at 30Hz, but for some reason it isn't. My GPU isn't a bottleneck. The GPU work queued to the Device Context is still taking 28ms to drain, which is the same as the good capture. The GPU is also only being kept busy about 60% of the time, and there's no reason that the DCS.exe thread couldn't run more often (at least, probably not)
  4. In summary, the main DCS.exe thread could be running at 30Hz, and my GPU would be able to keep up, but for some reason it isn't, resulting in a very poor framerate

My CPU also certainly isn't a bottleneck. I don't have an explanation as to why each frame of the DCS.exe thread went from 16ms to 23ms, but that's still fast enough to run at 30fps. I also have plenty of spare cores to which the DCS.exe thread could be scheduled to and run during this, so that isn't a problem either:

Omp4aKr.png

 

In summary, it should be evident from these captures that DCS is more than capable of running this at a stable 30fps here, but isn't due to what appears to be a bug with the main DCS.exe thread literally deciding to do nothing and wait (lazy thread!). There is no CPU/GPU bottleneck.

DCS must have some way of throttling how frequently the main DCS.exe thread runs. If it ran as frequently as it could, then my GPU wouldn't be able to keep up. This is why it only runs at 30Hz instead of 45Hz in the first capture, because even though it could run at 45Hz, the GPU would fall behind.

So it must be doing something intelligent to throttle how often it runs.

It's my theory that these framerate drops in VR (consistent with the symptoms mentioned here) are due to a bug with how the DCS main thread is scheduled to run, that causes it to run less frequently than it should in certain scenarios, and thus may be a relatively simple fix and a huge stability boost to anyone experiencing this issue (including myself).

I'm happy to provide more info. I also have logs available, but I'll note that they look the same whether or not the issue is occurring, so I don't believe they'll do much good.

I also have some experience in this field and would be happy to sign an NDA and look into the issue myself, but I'm sure ED has people better suited to fixing this than me. If only they're able to reproduce and look into the issue, I'm sure a lot of VR customers would be very grateful, as this issue is extremely disruptive, to the point of making the game unplayable in VR without excessive hardware purchases (like a 3090) or turning settings down to very low levels and avoiding maps like Marianas or even MP entirely.


Edited by muelch
typo
  • Like 2
  • Thanks 11
Link to comment
Share on other sites

Man ! This report is amazing ! Thanks a lot for all the effort you put in it. Very informative.

I too have experienced often those phenomenon. It seems to be triggered by high VRAM or RAM usage. but still very elusive to catch.

I hope that ED will look into it.

Link to comment
Share on other sites

It seems like this has something to do with Motion Reprojection. If I disable motion reprojection in OpenXR tools for WMR, the issue seems to go away (albeit, my fps is higher, but it's much less smooth without motion reprojection kicking in).

It seems like when I was testing through SteamVR earlier, it was still using WMR's motion reprojection, hence why it had the same issue.

As a workaround, I am using SteamVR as the OpenXR runtime, that way I can use SteamVR's motion smoothing as they call it. This seems to fix any prolonged FPS drops, with the downside that motion smoothing only works at 45 fps and not 30 fps, as well as the general terribleness that is SteamVR with DCS (i.e. seizure inducing loading screens). I also need to set "Force DX11 Mode" in WMR for SteamVR settings (along side where the motion projection setting is). My understanding is this has nothing to do with DCS (as DCS is already DX11), but affects how SteamVR is doing the motion reprojection (i.e. it will use DX11 instead of DX12). It may affect more than just the motion projection as well, things did seem a tad more blurry.

I'm assuming there's some kind of issue between DCS and the motion reprojection under DX12 (as the issue seems to occur with both SteamVR's motion reprojection under DX12, and WMR's motion reprojection, which is pretty curious). It seems like something is leaking, causing the motion reprojection on the GPU to have to do a lot more work, which is why framerate remains low in the main menu when this issue occurs.

It could also be totally unrelated to motion reprojection, and it just so happens that switching to SteamVR's motion reprojection, and forcing it to DX11, freed up enough vram to not trip the same issue.

The capture I posted earlier where DCS was seemingly able to run at 30fps but chose not to may still be a problem, but I don't think it's the main issue here.

Link to comment
Share on other sites

Hi guys, and thank you muelch for this amazing work.

Like mentioned in a different thread I ( and a few others) have had good results caping the RAM with memory cleaner

https://www.koshyjohn.com/software/memclean/

All I need to do is running this app in the backgroupnf and check in the options "staying under 80% of RAM",  not too sure how it can help, but definitely worth trying

 

Link to comment
Share on other sites

That could really only help if you're low on physical RAM (and even then I'm skeptical), which should be easy to spot (and probably best solved by fixing whatever is using so much).

I don't think anything that fixes could be consistent with the symptoms here (VR-specific, sometimes fixed by alt-tabbing, returning to main menu still seeing low FPS).

Link to comment
Share on other sites

I was able to confirm the same issue in MSFS. Triggering low fps in-game then carried back over to the main menu. Alt-tabbing fixed it.

Disabling motion reprojection also caused the issue to go away.

So seems like a really unfortunate bug with WMR's motion reprojection, possibly related to the RTX 3080, Reverb G2, and/or DX12.

 

Link to comment
Share on other sites

I'm having exactly the same issues the initial post describes: the drop in fps persisting to the main menu, the alt+tab sometimes fixing it, etc.

I'm using a RTX 3080 too, with a Reverb G2 and OpenXR, and disabled motion reprojection. 

Link to comment
Share on other sites

You're seeing this issue with motion reprojection disabled? That's a bit different, since I was only able to repro with motion reprojection enabled.

It also seems that enabling Hardware Accelerated GPU Scheduling makes the issue go away.

However, I saw a noticeable GPU frame time hit when enabling that, but only if motion reprojection wasn't enabled.

So if you aren't running with motion reprojection, you might try enabling Hardware Accelerated GPU Scheduling if it's not already and seeing if that makes the issue go away.

Link to comment
Share on other sites

I would rule out the reprojection part, as I have your exact same symptoms and I always have it disabled. Even the weird stuff like the frame drop continuing in the menu and alt-tab fixing it.

Also, today I tested enabling the hardware accelerated GPU scheduling, but I still got one occasion of frames dropping and continuing to be at 45 (instead of the normal 90) in the menu. So it didn't solve it 😞

Link to comment
Share on other sites

Yeah I was finally able to repro without reprojection. At least that rules one thing out.

I still have no luck reproducing with HAGS. Probably just masking whatever the issue really is on my machine.

Link to comment
Share on other sites

Okay, I think what's going on here is VRAM pressure is causing some critical VR-related resource(s) to get moved to shared GPU memory (i.e. system memory), which then tanks framerates since system memory latency is so much higher than VRAM latency.

You can see when this issue occurs after returning to the main menu, the GPU bus usage is incredibly high (top right, circled in red as it drops due to alt-tabbing):

j3llxpM.png

I circled in red some relevant metrics when alt-tabbing to fix the issue while being in the main menu when this issue is occurring.

Before alt-tabbing, GPU bus usage is roughly 40% (40% of what I'm not sure). After alt-tabbing, it goes down to 0%. Which makes sense, there's nothing on the main menu that should need to constantly transfer data between the CPU and GPU, better yet at the insane rate of 40% of the bus.

However, the reason this bug is happening is clearly because something is stuck in shared GPU memory that shouldn't be, even after returning to the main menu when we have plenty of dedicated GPU memory available, and it's probably needing to be transferred between the CPU and GPU every frame, perhaps frame buffers used for VR or something.

You can also see circled in red an increase in dedicated GPU memory usage when I alt-tab, and a decrease in shared GPU memory usage, as the issue fixes itself.

Something is getting moved from shared to dedicated GPU memory when alt-tabbing, which is why the issue fixes itself. I'm guessing alt-tabbing causes windows to tell the process to trim GPU resources, since it's no longer in the foreground and therefore shouldn't have as high of priority.

I haven't been able to track down what exactly the resource is.

I'm going to give ED the benefit of the doubt and assume it's something related to WMR/openxr, especially since I was able to reproduce this issue in MSFS. And I think it makes sense that it would be a large VR-related resource.

I think part of the issue is that there are two competing things going on here, DCS and the WMR/VR stuff. DCS runs in DX11, but even within DCS.exe, winxrruntime.dll is being used for VR, and it uses DX12.

In DX11, I believe it would semi-intelligently manage these resources, and ensure that whatever this high priority thing is that's ending up in shared GPU memory would instead be in dedicated, or at least get moved there when returning to the main menu.

In DX12, resources are managed manually, and it's the responsibility of applications to stay within their given budget to avoid stepping on other simultaneously running applications.

I'm not sure how this works with both DX11 DCS stuff and DX12 VR stuff running in the same process.

I'm also not sure if this is something that needs to be fixed by ED, Microsoft, or even possibly nvidia. E.g. maybe DCS needs to obey a stricter vram limit somewhere when in VR, or maybe Microsoft needs to pin these VR-related sources so they stay in dedicated gpu memory, or maybe this is the responsibility of the GPU driver.

Either way, I think the one thing that isn't a solution here is buying a video card with more vram. An RTX 3080 is more than capable of running this in VR, this is just an unfortunate software problem getting in the way.

The issue should be relatively easy to fix, but unfortunately requires someone to take ownership of it between ED/Microsoft/Nvidia.

  • Like 2
  • Thanks 1
Link to comment
Share on other sites

On 6/7/2022 at 9:18 PM, muelch said:

Link to MS ticket if anyone wants to upvote it:
https://aka.ms/AAh4zmi

"cannot open the page because the address is invalid."

Im having similar issues. Running great and then a big fps dip (10-14fps) for many seconds. I upgraded my cpu from 5600x to 5800X3D and it still happens (albeit for a much shorter period). Turned off fTPM on bios too to no avail.

I am suspecting this to be caused by nvidia driver, as it seems to have started right after I updated from 496.xx to 512.95.

Can anyone confirm this happens with older gpu drivers?

5800X3D, 3080, 32GB 3200MHz, Reverb G2 with OpenXR + FSR and game still runs like crap.

Link to comment
Share on other sites

Yuuuup been experiencing this on and off since I started playing in VR. Used to think it was vram related, then thought maybe sound, so perhaps it was vram after all.

Recently I have found immediately sending the Q2 to sleep the moment the FPS tanks (it goes down from 40 reprojected to 80, to stuck at 20 with no reprojection) for about 30 seconds fixed it. Happened twice today when I opened F10 in the hind, and it happens to me a LOT more in the hind than in other modules, which is why I used to think it was a vram issue.

Oh, and when it happens on the Q2 this starts getting spammed in the OVR server console: 

 

 

encoder backup.png

This only happens in DCS, never had it happen in any other VR game. Used to have to always restart DCS to clear it, but like I said the last two times the moment it happened I sent the Q2 to sleep for about 30 secs, woke it up and it was clear. Perhaps that is clearing the vram, as nothing is on display while it's asleep.

Still annoying as hell, and while bringing up F10 is one way to trigger the event it's far from the only one.


Edited by MoleUK
Link to comment
Share on other sites

I have somehow similar experience in VR using G2 with 3080ti and i7 and DCS OB. I am flying same mission in Caucasus using every helicopter. With the u-1h, mi-8, ka-50, gazelle and uh-60 the flight is smooth with ~45fps (22ms GPU time). When I switch to ah-64d or mi-24 flying the same route I experience 4-10fps with GPU time of 100-150ms!

I have been capturing logs and performance stats with no meaningful explanation of the issue related to CPU or GPU bottleneck but I have managed to reproduce with the most recent helicopters.

You can try with the attache track file and let me know.

VR_test.trk

Link to comment
Share on other sites

It's not module specific. The Apache and the Hind are just hardware heavy modules, mainly because they are new, with new graphics standards, probably not fully optimized yet etc. I've tried again to use the motion reprojection and that drop occured again. First I took the Hornet for a ride and it was all good. As soon as I switched to the Viper, my FPS went down to 15, which, obviously, was totally unplayable. It happens with the motion reprojection turned ON, and I believe with OpenXR only. Doesn't matter which module you use. Another case is on busy multiplayer servers, then the drop happens almost immediately, within 1 minute, not more.

Link to comment
Share on other sites

Apache and Hind cockpits both have 1 million triangles while, for example, Viper one has 350k, thats why those cockpits are so demanding

NZXT H9 Flow Black | Intel Core i5 13600KF OCed P5.6 E4.4 | Gigabyte Z790 Aorus Elite AX | G.Skill Trident Z5 Neo DDR5-6000 32GB C30 OCed 6600 C32 | nVidia GeForce RTX 4090 Founders Edition |  Western Digital SN770 2TB | Gigabyte GP-UD1000GM PG5 ATX 3.0 1000W | SteelSeries Apex 7 | Razer Viper Mini | SteelSeries Artics Nova 7 | LG OLED42C2 | Xiaomi P1 55"

Virpil T-50 CM2 Base + Thrustmaster Warthog Stick | WinWing Orion 2 F16EX Viper Throttle  | WinWing ICP | 3 x Thrustmaster MFD | Saitek Combat Rudder Pedals | Oculus Quest 2

DCS World | Persian Gulf | Syria | Flaming Cliff 3 | P-51D Mustang | Spitfire LF Mk. IX | Fw-109 A-8 | A-10C II Tank Killer | F/A-18C Hornet | F-14B Tomcat | F-16C Viper | F-15E Strike Eagle | M2000C | Ka-50 BlackShark III | Mi-24P Hind | AH-64D Apache | SuperCarrier

Link to comment
Share on other sites

2 hours ago, 5ephir0th said:

Apache and Hind cockpits both have 1 million triangles while, for example, Viper one has 350k, thats why those cockpits are so demanding

Agree on that but then mi-8 cockpit comes to my mind, which also seems like crazy complex.

Link to comment
Share on other sites

I'm having a similair issue where the fps tanks at random moments and causes massive stuttering with no escape. Restart of dcs is needed to get it fixed only for it to start again later. I can also reproduce it by going into a mission once, going out of it and then going to the settings menu. That causes the same drop which I cant escape from. 

Pretty frustrating. A bit tired of DCS right now because of it. 

Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...