Dedicated Server freeze

[mono6020]

Dedicated server freezes after running without any issues for days, when a user connects.

First it disappears from the server browser and after that, if you try to connect via IP, client just hangs at loading screen.

Taskmanager on the server then reports DCS.exe inactive, 27% CPU and 25GB RAM usage still counting up (32GB RAM available) 1MB per second. looks like there is still a serious memory leak.

lots of MOOSE scripting features and hundreds of dynamic spawns over a few days, but we always make sure every spawned unit gets despawned and cleaned up after landing or crash.

there is always only an amount of max. 50 AI units (air and ground combined) alive at any one time spread out over the whole PG Map. There are never more then 4 human players online at any time. Very small amount of ground units (~4) with dynamic pathfinding.

The freeze happens every time, with every mission somwhere around day 3 and it always starts with the following lines in dcs log:

 

2020-04-13 04:23:23.000 WARNING EDOBJECTS: RegMapStorage has no more IDs

2020-04-13 04:23:23.000 ERROR EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

 

and later it spams this lines in log:

 

2020-04-13 05:25:52.039 INFO SCRIPTING: stack traceback:

[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:22603: in function <[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:22599>

[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:105738: in function <[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:105725>

(tail call): ?

[C]: in function 'xpcall'

[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:22621: in function '_call_handler'

[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:22717: in function <[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:22627>

(tail call): ?

[C]: in function 'xpcall'

[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:5826: in function <[string "C:\Users\ds-admin\AppData\Local\Temp\2\DCS.openbeta_server\/~mis0000693C.lua"]:5770>

 

no ACCESS VIOLATION and no hard crash, the server just freezes and refuses to accept connections.

 

i really just want to know where the problem is to try and workaround. is it a DCS thing or a MOOSE thing? Do we hit any hardcoded limits? can i do anything to help fixing this bug? as you can surely imagine it's extremly frustrating to loose a 3 day mission progress and start all over again. Yes we use some harmless custom static object mods but as said on top of the post, the mission runs perfectly fine for the first 2 days. i dont think it's a scripting issue as well.

 

Full Log:

https://www.dropbox.com/s/p1n5wqdjc2l7cr4/dcs%20%281%29.log?dl=0

[Wesjet]

We are seeing some of the same error messages:

 

2020-04-18 04:17:44.868 ERROR Lua::Config: Call error debriefing.addEvent:[string "-- debriefing module events hook..."]:92: Parameter #1 (unit name string) missed

stack traceback:

[C]: ?

[C]: in function 'getByName'

[string "-- debriefing module events hook..."]:92: in function <[string "-- debriefing module events hook..."]:51>.

 

The above appears in pretty well every mission we run to a varying degree, and I am unable to tell if it is harmless in terms of the server operation, as it is apparent near the end of the log if the server crashed but other times the server keeps on running with many of these within a short time span but not consecutively.

 

One of our WWII missions is having some performance issues that may be tied to the issues with the ground units and pathing, however it is seeing the following error messages pretty well spammed into the log growing to over 90,000 lines in relatively short time period. Our other missions don't seem to show this error message type:

 

2020-04-18 04:17:46.248 WARNING EDOBJECTS: RegMapStorage has no more IDs

2020-04-18 04:17:46.248 ERROR EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

 

I cannot include the track file for the flight, as it is over 100MB for roughly 2 hours and 20 minutes of action. The partial server log is attached - up until I disconnected.

dcs - Copy.zip

[Pikey]

Folks, the error

2020-04-18 04:17:46.248 WARNING EDOBJECTS: RegMapStorage has no more IDs
2020-04-18 04:17:46.248 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42     

is caused when DCS reaches a hard limit for ID for new groups. I've attached a mission which explains the 4092 limit to group spawns. That's the cause of one issue but not quite the root cause of why the excessive spamming occurs.

 

From the OP's massive log full of errors, the server was up for three days constantly spawning aircraft as far as I can see. After a couple of days it ran out of ID's and began filling with errors. The reason for that I can't quite tell looks like a script was misconfigured - both Airboss and A2A_Dispatcher were RTBing aircraft in the same minute. Moose was also out of date. I don't know why they RTB'd immediately, it's not normal, the admin will have to figure that out by themself, it's not a DCS bug, it's a limit caused by a misconfiguration.

 

If you want to see the 4000 group limit in action, run the attached mission, so you know that I know what I'm talking about and read your logs and you will see it generate that error at 4000.

 

As for the second mission, not sure what that was all about, have no idea what was running on it, but it too ran out of ID's and worse.

 

Point of note. DCS is not designed for long running play and the expectation your server can stay up for 3 days is wrong. No, that is not written anywhere else that i know of, but a history of crashng and hard limtis and the single sortie design legacy is why most sensible admins reboot their servers often, seek persistence methods and read their logs before issue occur. Case in point, the kings of tuning, the Buddyspike team long discovered a 3-4 hour intense MP mission was about as much as was reliable. My own server can make 12 hours most good months depending on the patch. YMMV.

nomoreIDTest.miz

[mono6020]

Folks, the error

2020-04-18 04:17:46.248 WARNING EDOBJECTS: RegMapStorage has no more IDs
2020-04-18 04:17:46.248 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42     

is caused when DCS reaches a hard limit for ID for new groups. I've attached a mission which explains the 4092 limit to group spawns. That's the cause of one issue but not quite the root cause of why the excessive spamming occurs.

 

From the OP's massive log full of errors, the server was up for three days constantly spawning aircraft as far as I can see. After a couple of days it ran out of ID's and began filling with errors. The reason for that I can't quite tell looks like a script was misconfigured - both Airboss and A2A_Dispatcher were RTBing aircraft in the same minute. Moose was also out of date. I don't know why they RTB'd immediately, it's not normal, the admin will have to figure that out by themself, it's not a DCS bug, it's a limit caused by a misconfiguration.

 

If you want to see the 4000 group limit in action, run the attached mission, so you know that I know what I'm talking about and read your logs and you will see it generate that error at 4000.

 

As for the second mission, not sure what that was all about, have no idea what was running on it, but it too ran out of ID's and worse.

 

Point of note. DCS is not designed for long running play and the expectation your server can stay up for 3 days is wrong. No, that is not written anywhere else that i know of, but a history of crashng and hard limtis and the single sortie design legacy is why most sensible admins reboot their servers often, seek persistence methods and read their logs before issue occur. Case in point, the kings of tuning, the Buddyspike team long discovered a 3-4 hour intense MP mission was about as much as was reliable. My own server can make 12 hours most good months depending on the patch. YMMV.

 

thx for the info. will have a look at the scripts. at the date of post, moose was the latest dev version as far as i know (the one that works with 2.5.5 and 2.5.6). i can imagine it's not supported to have servers running that long. i'll try to reduce the amount of spawns maybe it helps, just wonder because the mission did indeed run for several days (way more then 3) a few months back with different dcs and moose version. will have a look where the mass spawns come from. makes no sense it spawns 4000+ units in 3 days

[Wesjet]

Pikey, thanks for the info!

I have asked my friend who designs the missions to check out the MOOSE version and see if we can try to log the group spawns. We try to cap missions with a 12 or 24 hour timeout, and then they automatically restart themselves via script. The server itself remains running and the total runtime of DCS.exe was measurable in months in some cases without incident. We have been fine with that method for all of the 2.5.5 branch.

 

The mission we were running was a WWII mission with back and forth AI ground and aircraft spawns that fly towards the AO and engage but it was not running nearly long enough for there to have been that many - not even 12 hours.