Crashing with DXGI_ERROR_DEVICE_REMOVED since 2.7.9

[BadgerFly]

Hi Everyone!

Since end December last year (since the OB 2.7.9 patch was released) I have sadly been getting repeated and repeatable crashes to desktop with the DXGI_ERROR_DEVICE_REMOVED every single time I fly. This has not always been the case.

 

After reading up on the error on the Eagle Dynamics and r/Hoggit forums, here is what I've tried so far:

- I am running WMR and SteamVR without the latest open betas.
- I have the latest nVidia drivers, my Windows is up to date and I have re-installed my nVidia drivers after using DDU. 
- My GPU Shader Cache is set to 10GB.
- My motherboard BIOS is running the latest version.
- I have tried running DCS Slow Repair, the sfc /scannow utility, and re-naming my Saved Games/DCS.Openbeta folder.
- I have tried underclocking my GPU, reducing the power limit to 80%, and running it in nVidia Debug Mode (lowers everything to stock settings).
- I have tried to swop around the 8-pin 12V power rails on my PSU.
- My CPU and GPU temps never exceed 65 - 70 degrees Celsius
- My Windows Pagefile is set to 64GB
- I have tried removing the only Mod I have installed, fholger's Reshade, and this has not made any difference.
- I have tried adding the TdrDelay and related registry keys into Windows.
- I have tried lowering all my graphics settings, but this has not helped either.

Essentially, what happens each time is that I load into DCS, load into a multiplayer server (or singleplayer, same thing happens), and load into an F-16, AV-8B or Ka-50, and after about 5-10 minutes of flying DCS crashes. Sometimes it crashes 'well' and writes a proper crash dump, other times it just crashes to desktop and writes to the DCS log.

 

I absolutely love DCS World and am incredibly grateful for all the amazing work that the ED team constantly does to make our dreams come true! I am simply getting desperate, since I am at the point of not being able to fly for more than 5-10 minutes before everything crashes.

Does anyone have any ideas of what else I could try, or do I have to face the fact that my graphics card could be on its last legs?

Thank you in advance for your time and help!

dcs.log

[Flappie]

I see you've already tried a lot of things that worked for some users.

Did you run a memory test?

[BadgerFly]

Hi Flappie

Thanks so much for the reply!

I have not tried that, no, can you recommend a suitable method to me / where can I find out how to do this? 

[maximov]

Goooooogle 
Or https://www.techpowerup.com/download/techpowerup-memtest64/
(for example). Also, a few questions:
1. Does the game crash without VR?
2. Does the game crash in the single player?
3. Does the game crash at the minimum graphics settings?

[Flappie]

Window also has a built-in utility called Memory diagnostic.

Edited January 10, 2022 by Flappie

[BadgerFly]

Hey Maximov, Flappie

Thank you for your replies! I'm currently at work, but I will see if I can run a memory test this evening, and also do some more testing so I can get you some more answers.

Thanks again!

[BadgerFly]

Hi Maximov, Flappie

So I ran the TechPowerUp memory test, and after 20 minutes and 6 loops it showed 0 errors, so that's at least one thing ruled out. 

To answer Maximov's questions:

1. The crashes do not happen in 2D with the same settings.
2. The crashes do happen in Singleplayer after about 5-10 minutes (not sure if I should be trying with a pre-defined mission, I'm just flying around after creating a basic mission in the ME).
3. I set the game to the default "Low" settings, and I did not experience any crashes. My FPS shot up to 65, but I could almost not make out what was going on on the HUD! 

So it seems from these questions that my system (or at least, my GPU) cannot handle very high settings - I suspect the crux of the matter might lie in my desire to have Textures set to high. My question is then: If I am happy to fly with 40-42 FPS in VR with High textures, why does my system crash? I mean, it clearly handles my desired settings for around 5 minutes or so, but then suddenly decides that it's had enough!

Do you think that I'm just being too optimistic in terms of what my 1080Ti can reliably sustain? And then, why has this changed with the recent update to 2.7.9? Can it be possible that my GPU might be giving up the ghost?

Thank you for your time.  

 

(I have attached my settings below. I also use SteamVR SuperSampling at 160%)

[maximov]

58 minutes ago, BadgerFly said:

so that's at least one thing ruled out.

No. Not yet. The test may not show an error. What frequency should your RAM have? What frequency do you actually have? What is your timing profile (BIOS)? You did the test, but we don't know anything.
We have too little data. You have not sent a single crash report to our server since January 1. There are no dcs.logs here either.

58 minutes ago, BadgerFly said:

it clearly handles my desired settings for around 5 minutes or so, but then suddenly decides that it's had enough!

It is typical for an old graphics card to overheat due to poor contact between the video chip and the heatsink. What is the temperature of the video chip when playing DCS? What is the core frequency and what is the video memory frequency?

[Raven (Elysian Angel)]

I'd suggest running a VRAM benchmark. DCS crashing in VR but not in pancake might be because VR uses quite a bit more VRAM (also depends on your resolution and SteamSS settings).
I had the exact same problem as you with my 1080Ti a few years back, and it turned out to be hardware failure (of the VRAM) - it started quite slowly but got worse over the period of a year until my card finally died.

[BadgerFly]

1 hour ago, Raven (Elysian Angel) said:

I'd suggest running a VRAM benchmark. DCS crashing in VR but not in pancake might be because VR uses quite a bit more VRAM (also depends on your resolution and SteamSS settings).
I had the exact same problem as you with my 1080Ti a few years back, and it turned out to be hardware failure (of the VRAM) - it started quite slowly but got worse over the period of a year until my card finally died.

Hi Raven

Thank you for your suggestion! Can you please recommend a VRAM benchmark for me? I've tried several, and I can't seem to get one to work

- OCCT tells me I don't have an OpenCL compatible device, 

- memtestG80 tells me I don't have a CUDA device

- Video Memory Stress Test just won't run. 

 

1 hour ago, maximov said:

No. Not yet. The test may not show an error. What frequency should your RAM have? What frequency do you actually have? What is your timing profile (BIOS)? You did the test, but we don't know anything.
We have too little data. You have not sent a single crash report to our server since January 1. There are no dcs.logs here either.

It is typical for an old graphics card to overheat due to poor contact between the video chip and the heatsink. What is the temperature of the video chip when playing DCS? What is the core frequency and what is the video memory frequency?

Ah, I see. My RAM is set at 3600MHz, Cl16, it is a G.Skill 2 x 16GB kit rated for this speed. 

Sadly, DCS has been crashing without being able to write crash logs, but I did attach a DCS log to my original post. 

Hmmmmm. Temperature could be a possibility - The card is around 4-5 years old now. FPS VR tells me that the card never exceeds 70C, but there might be a hotspot somewhere that's not being reported. The core and video memory frequency are now sitting at stock 1080Ti speeds - in the NVidia Control Panel I enabled 'Debug Mode' - this seems to drop the frequencies to stock accross the board. 

Thank you for your help, I will do some more testing on my own and play with some of the settings some more. 

[Raven (Elysian Angel)]

10 hours ago, BadgerFly said:

- OCCT tells me I don't have an OpenCL compatible device, 

- memtestG80 tells me I don't have a CUDA device

- Video Memory Stress Test just won't run. 

I'm far from a specialist, but perhaps that tells you something (providing your drivers are up to date and installed properly) ? I only used OCCT back then.
Perhaps you could borrow another card from a friend or someone you know, and see if that works?

[July]

I have been having the same exact issue, 1080 Ti and I think my card is failing. I've run memory diagnostics (like memtest86) I've tried downclocking/unclocking/undervolting everything just to try and rule out certain things like problems with my RAM or CPU (all diagnostics say they are OK/functional btw). I've done a factory reset of my PC and got a new NVME drive just trying to try and find a solution to this problem without actually buying a new GPU (which currently, is last resort for me) but lately I've been crashing with more frequency than ever and now it's not just limited to DCS, even Tarkov and Squad. Blue/Green/Pink flashes in the middle or corners of my screen, or crashes during GPU intensive events is probably a telling sign. Log is attached below just in case if someone asks.

dcs.log

Edited January 12, 2022 by July

[BadgerFly]

22 hours ago, Raven (Elysian Angel) said:

I'm far from a specialist, but perhaps that tells you something (providing your drivers are up to date and installed properly) ? I only used OCCT back then.
Perhaps you could borrow another card from a friend or someone you know, and see if that works?

Thank you Raven for your reply, I will see what I can get right with OCCT.

2 hours ago, July said:

I have been having the same exact issue, 1080 Ti and I think my card is failing. I've run memory diagnostics (like memtest86) I've tried downclocking/unclocking/undervolting everything just to try and rule out certain things like problems with my RAM or CPU (all diagnostics say they are OK/functional btw). I've done a factory reset of my PC and got a new NVME drive just trying to try and find a solution to this problem without actually buying a new GPU (which currently, is last resort for me) but lately I've been crashing with more frequency than ever and now it's not just limited to DCS, even Tarkov and Squad. Blue/Green/Pink flashes in the middle or corners of my screen, or crashes during GPU intensive events is probably a telling sign. Log is attached below just in case if someone asks.

dcs.log 117.6 kB · 0 downloads

 

Hey July, thank you for your post, it's bittersweet knowing someone else is also seeing the same thing happen. I reckon the writing is on the wall for me at least - it might be time to retire my 1080Ti. 

[maximov]

 July,  I have the same questions:
1. Does the game crash without TrackIR?
2. Does the game crash in single player?
3. Does the game crash when using the Caucasus map? 

" Blue/Green/Pink flashes in the middle or corners of my screen, or crashes during GPU intensive events is probably a telling sign"
Perhaps the video card is really going through the last days. But sometimes the video card simply does not have enough power from the power supply. And both of these components have been sucking dust into themselves for years. Overheating.

[maximov]

2 hours ago, BadgerFly said:

it might be time to retire my 1080Ti. 

Given the cost of video cards, I would recommend being fully convinced that the video card is faulty. I've had a couple of cases where a user bought a 3090 after our advice and got the exact same error. M-m-mm, we heard a lot of nice words. But after replacing the video card under warranty, the DCS worked though.

 

On 11.01.2022 at 13:05, Raven (Elysian Angel) said:

Perhaps you could borrow another card from a friend or someone you know, and see if that works?

This is ideal advice for your situation.

[July]

16 minutes ago, maximov said:

 July,  I have the same questions:
1. Does the game crash without TrackIR?
2. Does the game crash in single player?
3. Does the game crash when using the Caucasus map? 

1. Yes
2. Yes, but it takes quite a long time compared to crashes in multiplayer. Unless if the map is Syria, then it's fair game how long it takes to crash.
3. Yes

[maximov]

21 minutes ago, maximov said:

Perhaps the video card is really going through the last days

Nevertheless, attach a screenshot of your video card test when DCS is running. What is the frequency and temperature of the video core and video memory?
GPU-Z, MSI Afterburner...

[July]

9 minutes ago, maximov said:

Nevertheless, attach a screenshot of your video card test when DCS is running. What is the frequency and temperature of the video core and video memory?
GPU-Z, MSI Afterburner...

I will attempt to. I will add another thing though, these past several days my system has been going completely unresponsive and requiring a reset instead of only DCS crashing and producing a log. I may not be able to produce a screenshot when I crash that way, but I will try.

[July]

 

Here is a screenshot of conditions just before crashing, sometimes DCS will crash with a log, other times my entire PC will simply lock up and even waiting for 30 minutes does nothing. I end up having to reset. I have begun to experience this in other games as I stated before, similar conditions of +80% GPU usage into a lock-up. In these screenshots, I have underclocked my GPU core by 100mhz and memory clock by 300.

Edited January 12, 2022 by July

[MAXsenna]

For those of you that have issues, you should really try a GPU benchmark to see if it then crashes. If you're seeing artifacts, your GPU is about to die. Simple as that. Consider getting a heatgun and some high quality flux, and you might be able to save it when it goes. 

Cheers! 

[July]

11 minutes ago, MAXsenna said:

For those of you that have issues, you should really try a GPU benchmark to see if it then crashes. If you're seeing artifacts, your GPU is about to die. Simple as that. Consider getting a heatgun and some high quality flux, and you might be able to save it when it goes. 

Cheers! 

Any specific ones you can recommend or would something like running furmark for some time be good enough?

[MAXsenna]

4 minutes ago, July said:

Any specific ones you can recommend or would something like running furmark for some time be good enough?

Yup! That one and Unigine Heaven were the ones I used when one of mine decided to go. 

[maximov]

6 hours ago, July said:

In these screenshots, I have underclocked my GPU core by 100mhz and memory clock by 300.

1835-100=1735 MHz. But the reference frequency of the video core is only 1582 MHz in Boost Clock mode (!).
Then explain, are you setting records in overclocking or trying to work stable video card performance?

[July]

9 hours ago, maximov said:

1835-100=1735 MHz. But the reference frequency of the video core is only 1582 MHz in Boost Clock mode (!).
Then explain, are you setting records in overclocking or trying to work stable video card performance?

Huh, you're right. I didn't think much of it since I thought the regular boost clock went all the way up to 1950 but that clearly doesn't sound right. Maybe all this time I've been using my card with some kind of factory OC. To be clear, I have not tried to OC the card myself, and didn't realize that the regular core clock is supposed to be much lower than it shows. To be honest, I thought this was normal but maybe it has been being pushed to the point of instability. How would I disable the OC without manually clocking it down through something like afterburner? Through my Bios? Could this also be a product of Nvidia GPU boost without me having touched anything? https://www.nvidia.com/en-us/geforce/technologies/gpu-boost/technology/

 

Edit: A friend has asked me to try and lower the power and temp limits to try and lower the unstable boost clock as a result of the built in GPU boost, will report back when I have results

 

Edit 2: So far, so good, no significant crashes after downclocking my core to ~1650mhz and my memory clock to ~5000mhz and forcing my voltage lower, looks like GPU boost was causing instability. Running furmark for several minutes (~10 minutes) produced no noticable artifacting.

Edited January 13, 2022 by July

[BadgerFly]

Hey July

Something I found (which may or may not be a valid/good way to do this) is to enable Debug Mode in the NVidia Control Panel. From what I've seen, it essentially removes the OC - as soon as I enable it, Zotac Firestorm immediately shows the 1080Ti GPU clock decreasing to 1480MHz which I believe is the base clock for this card. 

So far, I've had limited positive results by enabling Debug Mode. As they say, your mileage may vary! I believe some more of my positive gains have been achieved by lowering my SteamVR Super Sampling - essentially lowering the workload on the GPU. 

(The screenshot below shows a Quadro P1000 - this is my work laptop)