Intertmittent MP Crash on Session End

[Panzertard]

I'm not sure if this is replicable, but I hope it's worth something for you at ED. Even if it's from a Windows 7 client.

Crashes does not happen with the Single Player engine, MP only.

And it's not that critical - it's managable to recover and start MP again.

 

Issue Synopsis:

Multiplay, after playing as client (only tested with "Near Gundelen - coop 2.miz"), when using Quit / Disconnect DCS may crash.

CTRL-ALT-DEL and Taskmanager is used to recover to desktop.

Windows Application Error reporting reports the error, and I've sent away a couple of them to Microsoft (if thats relevant for you).

EDIT: The crashes might occur in the middle of MP session as well as in sime single player situations.

 

Frequency: "Often" ( > 50%)

 

Client environment:

DCS:BS v.1.0, DVD UK/English - No addons / No LUA edits / No Skins

Windows 7 RC x64 edition.

Nvidia 295 GTX (896Mb Mem), 181.72 to 185.85

 

Components used besides DCS:

Saitek x52 Pro Stick (drivers only, no profile app installed)

Saitek Rudder Pedals (drivers only, no profile app installed)

Teamspeak 2

Teamspeak Overlay (Beta Build #63)

x52 GameControls Visible (mini-trick to enable more buttons for Saitek).

TrackIR 4 (v4.1.037)

 

Relevant incidents:

Saitek Rulez: I have to restart the services manually quite often, after game startup - this may lead to unhandled events/handles?

 

DxDiag xml, attached.

Crashlog attached.

 

Suspected cause:

I suspect there's a combination of software causing the issue. Not 100% sure on where/what so I will conduct further tests the following days.

 

Further tests will include (later):

* Disable some/all of the extra components used with DCS.

* Test other missions.

* Test being a server instead of a client.

 

If there are known issues I should be aware of, please guide me in the right direction. :)

DxDiag and Crashlog.zip

Edited June 6, 2009 by Panzertard
EDIT: The crashes might occur in the middle of MP session as well as in sime single player situations.

[Panzertard]

Oh, and I have a 21Mb "Ka-50\temp\CockpitCommandsLog.log" - is that normal? :)

[EtherealN]

That's interesting. My cockpitcommands.log is 1kb, containing only this:

 

Commands = {
{time = 0.000000,action = 2035,value  = 1.000000},
}

 

If you have somewhere to upload that file it might be enlightening to look at. (I've no clue based on what's already in there, but haven't looked at the crashlogs yet.)

 

Did send you some rep points for that awesome error report though. That's how it should be done man. :)

[Panzertard]

Hm, good point - I'll see if Im able to compress it to < 1mb (max rar/zip attachement size).

EDIT: Oh - Thanks for the +1 :)

 

EDIT2: Voila, 352Kb as rar. Attached.

CockpitCommandsLog.rar

Edited May 20, 2009 by Panzertard

[Panzertard]

Update #1

Frequency: ~100%, can replicate it mostly constantly at this computer.

Missions: Still present with other MP/Coop missions, including custom mission from the forum. No change.

Support software: Removed the TS Overlay software - no change.

Avoided to restart Saitek Service - no change.

 

Removed all Temp content.

CockpitCommandsLog still seem to be regenrated up into the > 20Mb size(s).

No change.

 

Eventlog, Application, Sample from tonight;

Faulting application name: DCS.exe, version: 1.0.0.0, time stamp: 0x494fc265

Faulting module name: ntdll.dll, version: 6.1.7100.0, time stamp: 0x49eea706

Exception code: 0xc0000005

Fault offset: 0x0002e1c3

Faulting process id: 0xf38

Faulting application start time: 0x01c9d99b3febfbb5

Faulting application path: D:\Games)\Eagle Dynamics\Ka-50\bin\stable\DCS.exe

Faulting module path: C:\Windows\SysWOW64\ntdll.dll

Report Id: 26c1d982-4597-11de-aa85-002354f2bfe3

 

Windows Error Report Summary, previous sample:

Description

Faulting Application Path: D:\Games)\Eagle Dynamics\Ka-50\bin\stable\DCS.exe

Problem signature

Problem Event Name: APPCRASH

Application Name: dcs.exe

Application Version: 1.0.0.0

Application Timestamp: 494fc265

Fault Module Name: MitkaGraphics.dll

Fault Module Version: 0.0.0.0

Fault Module Timestamp: 4933f4d6

Exception Code: c0000005

Exception Offset: 00029eba

OS Version: 6.1.7100.2.0.0.256.1

Locale ID: 1044

Additional Information 1: 29cf

Additional Information 2: 29cf22701e66cafdd15493b76e908f73

Additional Information 3: 6595

Additional Information 4: 6595ee09c3068edaa7d69334e0e6ecd2

Extra information about the problem

Bucket ID: 1246982989

 

Description

Faulting Application Path: D:\Games)\Eagle Dynamics\Ka-50\bin\stable\DCS.exe

Problem signature

Problem Event Name: APPCRASH

Application Name: dcs.exe

Application Version: 1.0.0.0

Application Timestamp: 494fc265

Fault Module Name: wRadio.dll

Fault Module Version: 0.0.0.0

Fault Module Timestamp: 4933f451

Exception Code: c0000005

Exception Offset: 000193fc

OS Version: 6.1.7100.2.0.0.256.1

Locale ID: 1044

Additional Information 1: a81d

Additional Information 2: a81d6df98e5795e99bdc93265c724fbf

Additional Information 3: 5bed

Additional Information 4: 5bed39351d76522648f9f3250116e7bd

Extra information about the problem

Bucket ID: 1217338143

Edited May 21, 2009 by Panzertard

[Panzertard]

Hm. Seeing the WOW64\ntdll.dll there in update #1 - this may indicate a 32->64 issue, compatability. Thats quite likely to appear for a 32 bit application, still, I might try some compatability modes on the DCS.exe instead.

 

And the references to the modules in DCS that may have been involved -

* Fault Module Name: MitkaGraphics.dll

* Fault Module Name: wRadio.dll

Does this mean it's graphic AND audio related?

ED, you may have some suggestions maybe? :)

Edited May 21, 2009 by Panzertard

[Panzertard]

Update #2 ("Near Gundelen - coop 2.miz")

* Shorter flights does not cause crashes.

* Shorter flights involving killing static objects - no crashes.

Edited May 21, 2009 by Panzertard

[Panzertard]

Update #3: ("Near Gundelen - coop 2.miz")

* Flying longer flights with TS / TS Overlay removed - no change, reproduced the crash.

* Flying out to mission area in and back - no change, reproduced the crash.

* Flying longer flights without engaging targets - no change, reproduced the crash.

 

Conclusion so far;

* TS or TS overlay doesn't affect the issue

* Flight length, landscape / number of textures / number of objects does affect the issue.

Edited May 21, 2009 by Panzertard

[c0ff]

Looks like DCS is hitting address space limit.

A tough problem. We know about the issue and working on it, but most certainly first patch will not fix it.

 

<spoiler>

There is a remote possibility of an experimental 64-bit build included into the patch.

It has much larger address space to fill :)

</spoiler>

Edited May 22, 2009 by c0ff
found .crash file

[Panzertard]

Thank you c0ff. And thank you for the update on your end, looking forward to the experimental build :)

Would it help if I ran some memory monitoring to confirm that this is the issue?

[c0ff]

Would it help if I ran some memory monitoring to confirm that this is the issue?

 

Sure.

[Panzertard]

Is there some parameters you are specifically loking for, or trends? Is there something that would be more to valuable include or monitor, in your perspective? :)

[c0ff]

http://hashpling.org/asm/

 

The final section has some textual summaries for the address space. The first

number is the total amount of committed address space for the whole process, the

second number is the total amount of reserved address space for the process, the

third number is the total amount of free memory and the final number is the size

of the largest free address space region.

 

Interesting to see the 4 numbers in case of crash (MP) and normal work (SP).

 

Thank you.

[Panzertard]

Cheers :thumbup:

Will conduct some more testing tomorrow.

[Panzertard]

I might have done something slightly lame regarding the consitency of reproduction of this issue - I upgraded my Nvidia drivers from 181.72 to 185.85.

And in the name of science and easier debugging I swithed to Windowed Mode.

 

Hence this update #4:

* Similar issue with singleplayer, after mission end (~1 hour flying, "Clear Tkvarcheli.miz") - about to exit, then game crash. (Crashlog attached)

I must admit I had no problems with this mission previously (2 times flown) - but I might have spent more time ingame this time. I will have to verify that it is possible to consistently reproduce this in the same manner as the MP type of crash.

 

Here's an illustration that includes the App Event Log and ASM output at the moment of impact.

 

My tests will continue, I Plan to set up the Performance monitor to monitor the DCS process with these parameters:

 

DCS-20090523-005931.rar

Edited May 23, 2009 by Panzertard
Attachment, crashlog

[Panzertard]

Update #5

ASM from a "good" working MP-session and end - ~1 hour flight "Ambush" mission.

Fullscreen mode, TrackIr, Teamspeak + Overlay and everything else running.

[c0ff]

The first number is used virtual address space. I think ASM does not take into account space used by the system (i.e. mapped graphical memory), so actual usage is much closer to the limit.

 

So, I think that any mission which takes >=1.5G in the first number will fail on exit.

 

Can you test this theory, please?

Hint: the more types of units mission has, the more address space it uses.

[Panzertard]

Certainly, we'll have a go tonight :)

 

EDIT: Placeholder for update#6 and the mission "Behind Enemy Lines"

Edited May 23, 2009 by Panzertard

[snooker]

Hi All,

 

I've got a similar problem.

My Black Shark always crash on Solo and MP session end.

 

I tried to setup Black Shark on Windows Vista 64b, Windows 7 32b and Windows 7 64b.

I tried the lower graphics quality.

 

The bug is present yet...

[EtherealN]

Snooker, as far as I know the adress space available to the application is not changed directly by which type of OS one runs. If the application is 32-bit then it's a 32-bit adress space, even if running on a 64bit OS. However, a 64bit windows does allow for running both 32 and 64 bit applications - something that a 32bit windows cannot do, ofc.

Edited June 3, 2009 by EtherealN

[Distiler]

Yes this happens to me when I fly large missions for lot of time. Seems ram fills too much because it crashes when I reach 1.4-1.5GB (I think 32bit applications can't handle more than 2GB, but maybe this is taking into account grpahic card memory that in my case is 512MB).

 

What I don't understand is how it can grow from 0.5GB at the mission start to 1.5GB. Is there a memory leack perhaps?

Edited June 3, 2009 by Distiler

[EtherealN]

Well, I would assume that it loads things a bit on-demand, so as new units are spawned or come in closer to you (causing meshes and textures to load on higher detail) and so on more space will be needed. And it's not always that these things can be unloaded safely since there might be more units of the same type out there.

[Panzertard]

Hi Distiler. Let's call this #update 6 - or the lack of.

Sorry. I've been somewhat "offset" regarding my perception of the cause of this - and how to consistently reproduce the problem.

 

- I've flown Coop on Quirks server for hours and hours (altough never with ASM loaded) - No crashes.

- I've tested loading other stuff - and "cheating to raise the memory" > 1.6Gb - using the F11 view (eats memory). No crashes.

Neither above isnt consistent with expected behaviour - should have crashed. Should.

And I've tested quite a few different situations, all with the memory in ASM > 1.55 by great - with lack of crash as a result.

Here's a sample (which had peeked at 1.68 before I was able to grab a screenshot):

 

 

I need to return and test the crash that was supposed to be consistent again.

Edited June 3, 2009 by Panzertard

[Panzertard]

update #7

Flying on 159's server - and huge mission(s) quite frequently brings more crashes.

Using ASM I was able to spot that I was close to 1.50GB consumption before one of the crashes.

Altough adding ASM to the sandwitch seem to create a bit more unstability (at least on two occasions).

 

Here's that crahs report:

# -------------- 20090606-005659 --------------
C:\Windows\SysWOW64\ntdll.dll
# C0000005 ACCESS_VIOLATION at 7713E1CE 01:0001E1CE
#
# EAX = 4B7A3C83
# EBX = D57D3C84
# ECX = 00003F7F
# EDX = 00003F73
# ESI = 224303F2
# EDI = 224303FA
# CS:EIP = 0023:7713E1CE
# SS:ESP = 002B:0018F88C  EBP = 0018F8C0
# DS = 002B  ES = 002B  FS = 0053  GS = 002B
# Flags = 00010202
7713E1CE 0018F8C0 0001:0001E1CE C:\Windows\SysWOW64\ntdll.dll RtlFreeHeap()+25E
7713E1CE 0018F8C0 0001:0001E1CE C:\Windows\SysWOW64\ntdll.dll RtlFreeHeap()+25E
7713DFEE 0018F8D8 0001:0001DFEE C:\Windows\SysWOW64\ntdll.dll RtlFreeHeap()+7E
76691488 0018F8EC 0001:00001488 C:\Windows\syswow64\kernel32.dll HeapFree()+14
7C34218A 0018F934 0001:0000118A D:\Games)\Eagle Dynamics\Ka-50\bin\stable\MSVCR71.dll free()+39
000314D3 0018F93C 0001:000104D3 D:\Games)\Eagle Dynamics\Ka-50\bin\stable\lua.dll luaL_loadstring()+53
00028D51 0018F960 0001:00007D51 D:\Games)\Eagle Dynamics\Ka-50\bin\stable\lua.dll luaM_realloc_()+21

 

The others are similar - same modules being present at the crash.

 

I was thinking a bit on the x86, 32 bits 2 gb adress space limitations thingy.

I've been in similar situation previously with games such as X3:Terran Conflict. Back then they lacked the ">2GB Adress space" flag on the Executable, altough it only cause some performance issue only.

DCS.exe has this flag enabled already from what I can see.

External reference to that type of issues: http://www.jaloonz.com/2008/04/enabling-large-address-aware-for-games.html

Asumming this is the flag: http://msdn.microsoft.com/en-us/library/aa366912(VS.85).aspx

 

I tried turning it off (so I could test if I could provoke another outcome / error) but of course copy protection wont allow me to test with a 2Gb adress limit, the file has been modified. :)

(I hope you wont come down on me for tampering with the file like this, c0ff) :)

 

Any other ideas or suggestion we can try?

Edited June 6, 2009 by Panzertard

[Panzertard]

Thinking again - too much pondering, but I cant let it go ;)

 

Looking at some other issues for Vista (both 32 and 64 editions) there seem to be some situations where you can exhaust your virtual adress pool.

I can't really tell if this is relevant for this case, but I'm sure ED might have someone capable of that. :)

If an application creates its own in-memory copy of its video resources, or the application uses DirectX 9 or an earlier version, the virtual address space contains the WDDM video memory manager's virtualized range and the application's copy. Applications that use graphics APIs that are earlier than DirectX 10 and that target GPUs that have large amounts of video memory can easily exhaust their virtual address space.

http://support.microsoft.com/kb/940105

 

Now also considering the address mapping from the Nvidia 295 GTX:

Dedicated Video memory: 896 Mb

Shared System Memory: 2811 Mb

Total Available Graphics mem: 3707 Mb

 

... not sure how this impacts the total mapping going on.