reported 2.7 multiplayer crash - possible related to eject and/ or map?

[philstyle]

Storm of War server, Normandy map.
Regular crashes without producing a crash dump or trk file. DCS just shuts down completely.
We are seeing similar patterns in the log each time, like this:

 

2021-04-25 13:07:24.058 WARNING EDOBJECTS: RegMapStorage::ForceID() - id is busy

2021-04-25 13:07:24.058 ERROR EDOBJECTS: Failed assert success at Projects\edObjects\Source\Registry\RegMapStorage.cpp:178

2021-04-25 13:07:24.058 INFO Scripting: event:type=eject,initiatorPilotName=ironbutterfly,target=,t=16949.728,initiatorMissionID=1695,targetMissionID=0,

 

Full log for reference:

https://drive.google.com/file/d/1sO2qJtMTmnWn61SBbr2ZZxewaY6Mogb-/view?usp=sharing

Edited April 25, 2021 by philstyle

[philstyle]

here is the trk from the player who recorded the "eject" in the final line of the above log:
https://drive.google.com/file/d/1C67dIzqtFXoO7e8fmWiBsWS4qlAsY2MJ/view?usp=sharing

[philstyle]

Another crash over night.

No mods running on server.  We're seeing the same lines in the log just prior to the crash over and over:

 

2021-04-25 23:02:29.007 WARNING EDOBJECTS: RegMapStorage::ForceID() - id is busy 2021-04-25

23:02:29.007 ERROR EDOBJECTS: Failed assert success at Projects\edObjects\Source\Registry\RegMapStorage.cpp:178 2021-04-25

23:02:29.007 INFO Scripting: event:type=eject,initiatorPilotName=SKettshy,target=,t=12177.307,initiatorMissionID=2447,targetMissionID=0,

Full log here:
https://drive.google.com/file/d/18x_xE2_mT-31fYTxqVuMmB1flQiP1MhW/view?usp=sharing

I reiterate that DCS is closing instantly without producing a crash dump file.

Edited April 26, 2021 by philstyle

[philstyle]

this could be a new crash, am not sure.
The content itself is different from the above, although same server.

 

Logs and crash dump files were generated this time.
They are all stored here:

https://drive.google.com/drive/folders/17A3uNubPjxKadeyZ-4QuoJvLfG3_W4Yi

Edited April 26, 2021 by philstyle

[BIGNEWY]

thanks I have shared in our report

[philstyle]

After a few hours people joining the server are getting a join-crash at the  "Net Sim Post Start" phase of joining.
A a screenshot is attached.

 

[Special K]

philstyle reported an issue lately this evening, when people that try to join got a client crash. Nobody was ably to join anymore.

We have the crash logs from 2 ppl, both of them have the exact same trace in it, so here is just one:

 

2021-04-26 21:16:24.962 INFO    EDCORE: # -------------- 20210426-211625 --------------
2021-04-26 21:16:24.963 INFO    EDCORE: DCS/2.7.0.5118 (x86_64; Windows NT 10.0.19041)
2021-04-26 21:16:24.964 INFO    EDCORE: C:\Steam\steamapps\common\DCSWorld\bin\DCS.exe
2021-04-26 21:16:24.965 INFO    EDCORE: # C0000005 ACCESS_VIOLATION at 07F65FBB 00:00000000
2021-04-26 21:16:24.969 INFO    EDCORE: SymInit: Symbol-SearchPath: '.;C:\Steam\steamapps\common\DCSWorld;C:\Steam\steamapps\common\DCSWorld\bin;C:\WINDOWS;C:\WINDOWS\system32;SRV*C:\websymbols*https://msdl.microsoft.com/download/symbols;', symOptions: 530, UserName: 'user0'
2021-04-26 21:16:24.970 INFO    EDCORE: OS-Version: 10.0.19041 () 0x300-0x1
2021-04-26 21:16:25.701 INFO    EDCORE: 0x00000000007B5FBB (DCS): SW + 0x4C560B
2021-04-26 21:16:25.702 INFO    EDCORE: 0x00000000008EF8AD (DCS): SW + 0x5FEEFD
2021-04-26 21:16:25.702 INFO    EDCORE: 0x000000000093508C (DCS): SW + 0x6446DC
2021-04-26 21:16:25.702 INFO    EDCORE: 0x0000000000939D88 (DCS): SW + 0x6493D8
2021-04-26 21:16:25.702 INFO    EDCORE: 0x00000000000A6CE1 (WorldGeneral): slot_decompose_ex + 0xCF1
2021-04-26 21:16:25.703 INFO    EDCORE: 0x00000000000AADE7 (WorldGeneral): WorldManager::notifyBirth + 0x37
2021-04-26 21:16:25.703 INFO    EDCORE: 0x00000000000CD786 (Weapons): wBomb::netCreate + 0x2B6
2021-04-26 21:16:25.703 INFO    EDCORE: 0x00000000000DD274 (Weapons): weaponInitRandomSeed + 0x1494
2021-04-26 21:16:25.703 INFO    EDCORE: 0x0000000000100743 (edCore): Mail::Reader::stream + 0x1473
2021-04-26 21:16:25.703 INFO    EDCORE: 0x00000000009A4D4F (DCS): SW + 0x6B439F
2021-04-26 21:16:25.703 INFO    EDCORE: 0x00000000009A5011 (DCS): SW + 0x6B4661
2021-04-26 21:16:25.703 INFO    EDCORE: 0x00000000009D2720 (DCS): SW + 0x6E1D70
2021-04-26 21:16:25.703 INFO    EDCORE: 0x00000000009B96E9 (DCS): SW + 0x6C8D39
2021-04-26 21:16:25.703 INFO    EDCORE: 0x000000000098F7C2 (DCS): SW + 0x69EE12
2021-04-26 21:16:25.704 INFO    EDCORE: 0x00000000007FF2C0 (DCS): SW + 0x50E910
2021-04-26 21:16:25.704 INFO    EDCORE: 0x00000000007D02A4 (DCS): SW + 0x4DF8F4
2021-04-26 21:16:25.704 INFO    EDCORE: 0x00000000007D067D (DCS): SW + 0x4DFCCD
2021-04-26 21:16:25.704 INFO    EDCORE: 0x0000000001C2ACAB (DCS): AmdPowerXpressRequestHighPerformance + 0xE3FCA7
2021-04-26 21:16:25.704 INFO    EDCORE: 0x0000000000A067DE (DCS): SW + 0x715E2E
2021-04-26 21:16:25.704 INFO    EDCORE: 0x0000000000017034 (KERNEL32): BaseThreadInitThunk + 0x14
2021-04-26 21:16:25.704 INFO    EDCORE: 0x0000000000052651 (ntdll): RtlUserThreadStart + 0x21
2021-04-26 21:16:26.218 INFO    EDCORE: Minidump created.
 

---------

Server side looks good so far (times are a bit off):

2021-04-26 21:27:17.020 INFO    NET: accepting connection from 188.194.226.232:53311
2021-04-26 21:27:17.033 INFO    NET: adding client[7]
2021-04-26 21:27:17.385 INFO    NET: added client[7] name=No.54 Philstyle addr=188.194.226.232:53311
2021-04-26 21:27:17.392 INFO    LuaNET: SLMOD INFO: connected client 7
2021-04-26 21:27:17.392 INFO    LuaNET: SLMOD INFO: num clients: 23
2021-04-26 21:27:18.472 INFO    NET: client[7] UDP port changed from 53311 to 53087
-----

 

As the user count dropped to about 10, people were able to connect again.

[davidp57]

I've already seen this RegMapStorage.cpp error in another older report ; at that time the dev team answered that this was a problem of map object count (too many, not enough ids)

[philstyle]

Another server crash just now.
No crash dump created, DCS just insta-closed.

Log file is here:

https://drive.google.com/file/d/1uSdgwykP-_m60XG-ku0uPxSe1OfqfU6G/view?usp=sharing

 

The last few lines are the same thing we've been seeing over and over for days now, and have been posted here numerous times:

2021-04-27 18:22:59.532 WARNING EDOBJECTS: RegMapStorage::ForceID() - id is busy
2021-04-27 18:22:59.532 ERROR EDOBJECTS: Failed assert success at Projects\edObjects\Source\Registry\RegMapStorage.cpp:178
2021-04-27 18:22:59.532 INFO Scripting: event:type=eject,initiatorPilotName=JetRo,target=,t=17988.781,initiatorMissionID=1705,targetMissionID=0, 2021-04-27 18:22:59.532 INFO Scripting: event:targetPilotName=JetRo,type=kill,weapon=,target=P-51D,t=17988.781,initiatorMissionID=19178,targetMissionID=1705,

Edited April 27, 2021 by philstyle

[TFV - Shepard]

Out of curiosity, have you guys tried reinstalling DCS, what OS are you using, and are you using an AMD processor?

I ask because we have a mission running on our server (Windows 10, Ryzen 5 3600) that was crashing on ejects like your server, we stood up a server in OVH (Windows Server 2016, Xeon E-2136, fresh 2.7 install) and we put the same mission on that server with zero changes and have had zero crashes.

[philstyle]

15 minutes ago, TFV - Shepard said:

Out of curiosity, have you guys tried reinstalling DCS, what OS are you using, and are you using an AMD processor?

I ask because we have a mission running on our server (Windows 10, Ryzen 5 3600) that was crashing on ejects like your server, we stood up a server in OVH (Windows Server 2016, Xeon E-2136, fresh 2.7 install) and we put the same mission on that server with zero changes and have had zero crashes.

 

We are using Windows Server with AMD CPU yes.
Have done a couple of repairs, but at this point not a complete DCS reinstall . . .

Edited April 27, 2021 by philstyle

[TFV - Shepard]

9 minutes ago, philstyle said:

 

We are using Windows Server with AMD CPU yes.
Have done a couple of repairs, but at this point not a complete DCS reinstall . . .

 

A reinstall of 2.7 on our AMD box was insufficient *however* it was just running the uninstall and reinstalling. Didn't clear out the Saved Games folder or anything. Tomorrow we're going to install Windows Server (needs to be done anyways) which will have a completely fresh install of 2.7. I'll report back once that's all done (might take all week depending on our availability). 

If it stops crashing with a fresh install, I suspect there is an issue with pre-existing 2.5.6 files somewhere on the system. If it keeps crashing, I'm suspecting some esoteric issue with AMD processors.

I've attached one of the crash dumps for reference, but essentially, we get these which looks to be similar in to the events in the OP. Same mission, new server, these events don't happen at all. 

 

2021-04-25 09:18:02.181 INFO    Scripting: event:type=eject,target=,t=11414.025,initiatorMissionID=1000167,targetMissionID=0,
2021-04-25 09:18:02.181 ERROR   SCRIPTING: Mission script error: [string "C:\Users\CSG-8S~1\AppData\Local\Temp\DCS.openbeta_server\/~mis00003768.lua"]:8252: Unit doesn't exist
stack traceback:
	[C]: ?
	[C]: in function 'getName'
	[string "C:\Users\CSG-8S~1\AppData\Local\Temp\DCS.openbeta_server\/~mis00003768.lua"]:8252: in function 'onEvent'
	[string "Scripts/World/EventHandlers.lua"]:13: in function <[string "Scripts/World/EventHandlers.lua"]:11>

 

dcs.20210425-092029.log

[davidp57]

at that point you should look in the referenced file :

C:\Users\CSG-8S~1\AppData\Local\Temp\DCS.openbeta_server\/~mis00003768.lua

[philstyle]

Made it through 24 hours this time before we had a crash. Crash occurred once player count had been over 35 for around 45 minutes.

We have enabled full data logging now.

The log is showing the same few lines right before the crash that we've been seeing over and over again.
namely:

2021-04-28 16:57:39.424 WARNING EDOBJECTS: RegMapStorage::ForceID() - id is busy
2021-04-28 16:57:39.424 ERROR   EDOBJECTS: Failed assert `success` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:178

 

Full log here:
https://drive.google.com/file/d/18ENrJRHRFNAGOtXo7OXObKWMhBSjlc1j/view?usp=sharing

 

Last few lines:

 

2021-04-28 16:57:36.433 DEBUG   Scripting: 	 ATCToLeaderHandler : make, coalition = 1
2021-04-28 16:57:36.433 DEBUG   Scripting: Phrases make : key = 4
2021-04-28 16:57:36.433 DEBUG   Scripting: Phrases make : key = 1
2021-04-28 16:57:36.433 DEBUG   Scripting: Phrases make : key = 4274
2021-04-28 16:57:36.433 DEBUG   Scripting: Phrases make : key = 0
2021-04-28 16:57:37.441 INFO    Scripting: event:targetPilotName=Paranoid,type=start shooting,weapon=Bofors 40mm gun,target=FW-190D9,t=17121.155,initiatorMissionID=1549,targetMissionID=18823,
2021-04-28 16:57:37.441 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17121.155,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:37.954 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17121.659,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:38.420 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17122.137,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:38.928 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17122.645,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:39.410 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17123.124,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:39.424 WARNING EDOBJECTS: RegMapStorage::ForceID() - id is busy
2021-04-28 16:57:39.424 ERROR   EDOBJECTS: Failed assert `success` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:178
2021-04-28 16:57:39.424 INFO    Scripting: event:type=eject,initiatorPilotName=Paranoid,target=,t=17123.131,initiatorMissionID=18823,targetMissionID=0,
2021-04-28 16:57:39.424 INFO    Scripting: event:targetPilotName=Paranoid,type=kill,initiatorPilotName=Willzah,weapon=,target=FW-190D9,t=17123.131,initiatorMissionID=18890,targetMissionID=18823,
2021-04-28 16:57:39.912 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17123.624,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:39.925 INFO    Scripting: event:weapon=Bofors 40mm gun,t=17123.624,type=end shooting,initiatorMissionID=1549,
2021-04-28 16:57:40.760 INFO    Scripting: event:targetPilotName=Paranoid,type=start shooting,weapon=Bofors 40mm gun,target=FW-190D9,t=17124.475,initiatorMissionID=1549,targetMissionID=18823,
2021-04-28 16:57:40.760 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17124.475,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:41.282 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17124.978,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:41.583 INFO    Scripting: event:type=hit,weapon=Bofors 40mm HE,target=,t=17125.294,initiatorMissionID=3169,targetMissionID=,
2021-04-28 16:57:41.783 WARNING LOG: 5 duplicate message(s) skipped.
2021-04-28 16:57:41.783 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17125.486,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:42.285 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17125.993,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:42.787 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17126.498,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:43.289 INFO    Scripting: event:weapon=Bofors 40mm HE,t=17126.999,type=shot,initiatorMissionID=1549,
2021-04-28 16:57:43.303 INFO    Scripting: event:weapon=Bofors 40mm gun,t=17126.999,type=end shooting,initiatorMissionID=1549,

 

Edited April 28, 2021 by philstyle

[philstyle]

Yet another crash overnight. Exactly the same lines, once again ion the log immediately prior to the crash:
 

Quote

 

2021-04-28 22:34:25.160 WARNING EDOBJECTS: RegMapStorage::ForceID() - id is busy

2021-04-28 22:34:25.160 ERROR   EDOBJECTS: Failed assert `success` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:178

2021-04-28 22:34:25.160 INFO    Scripting: event:type=eject,initiatorPilotName=Upgrayedd 1-1,target=,t=18235.769,initiatorMissionID=18860,targetMissionID=0,

2021-04-28 22:34:25.160 INFO    Scripting: event:targetPilotName=Upgrayedd 1-1,type=kill,target=SpitfireLFMkIX,t=18235.769,weapon=Bf 109 K-4,targetMissionID=18860,

 

 

Full log here:
https://drive.google.com/file/d/1hfQ8sE1gA4kkJtr846lqBADlVYSkRhHW/view?usp=sharing

[philstyle]

In addition to the consistent log file entries every time the server crashes, we are also getting the same critical error in the windows event viewer. All of them related to a failure in C:\Windows\SYSTEM32\ntdll.dll

 

This dll is reported by windows as the faulting module.

 

Attached are screenshots of the last four crashes reported in the windows event viewer:

Edited April 29, 2021 by philstyle

[Nirvi]

server disconnect 25.04.2021

SoW - July 18 1944 - MORNING v96b-20210428-185525.trk

 

server disconnect 28.04.2021

SoW - June 12 1944 - AFTERNOON v96b-20210425-133258.trk

[philstyle]

SoW crash again this evening.
Crash dump files and server log available from link below. Goes without saying, exactly the same errors / lines reported that we've been seeing for more than a week.

log: https://drive.google.com/file/d/15cegLjflI9xp9-mNx3HbBO-6jEpRDzRO/view?usp=sharing

crash & dump files:
https://drive.google.com/file/d/1bdvLbj9Bf_JJHjmeyn_-kGMsMAHEDi9o/view?usp=sharing
https://drive.google.com/file/d/14sycmUyNprDInIRa1PMvTHVQE7xHtYFq/view?usp=sharing

 

player trk file from a player who was online at the time of the crash:
https://wetransfer.com/downloads/a6a4feabec5e4433460eb69743f1726920210430190725/37a23f

This will be the last crash report from us on this build. We are no longer learning anything new or receiving any new information that might help.

[Gili]

Here as well my track file from the 28.04.2021 when the server disconnected. 

 

https://drive.google.com/file/d/1dc0pBmh0vPEtJFUOkMIUxKtirY0KReY2/view?usp=sharing

 

 

Edited May 2, 2021 by Gili

[LeLv8_Archi]

Server disconnect 3.5.2021

 

https://1drv.ms/u/s!At6da_1u-CIDgq1rNQsLFZHcEKDvnw

[BIGNEWY]

Hi all

 

just a little progress to report, we are testing a potential fix, if it is successful we will let you know. 

 

thanks

[philstyle]

We are happy that this is resolved now.
Thanks ED! Feel free to close this thread.