[REPORTED]2.5.6 Dedicated server mission High Mem usage

[wopski]

High there our dedicated server just decided to use 25GB of memory and mega FPS spikes.

 

Same mission pre 2.5.6 was 4 to 6 GB? ?

 

seeing this message in server logs

 

2:51:43 PM.400 ERROR DCS Mission script error: : [string "C:\Users\root\AppData\Local\Temp\2\S1\/~mis00006363.lua"]:6997: attempt to index local 'EventMeta' (a nil value) stack traceback: [C]: ? [string "C:\Users\root\AppData\Local\Temp\2\S1\/~mis00006363.lua"]:6997: in function 'onEvent' [string "Scripts/World/EventHandlers.lua"]:13: in function <[string "Scripts/World/EventHandlers.lua"]:11>

2:52:05 PM.760 ERROR DCS Mission script error: : [string "C:\Users\root\AppData\Local\Temp\2\S1\/~mis00006363.lua"]:6997: attempt to index local 'EventMeta' (a nil value) stack traceback: [C]: ? [string "C:\Users\root\AppData\Local\Temp\2\S1\/~mis00006363.lua"]:6997: in function 'onEvent' [string "Scripts/World/EventHandlers.lua"]:13: in function <[string "Scripts/World/EventHandlers.lua"]:11>

[Maverick87Shaka]

We also have a lot of error with our mission, with a massive memory leak in a couple of minutes that make server unstable or crashing to be run out of memory

I don't have the chance to test it with a "clean" without script mission, but i suspect there was a problem also with native mission editor triggers etc.

[lambo93]

We are also experiencing this issue too! The server is lucky to stay up for more than 20 mins before the stuttering and crash.

Also noticed the really high memory usage too!

[Ziptie]

If you are using a Stennis - try removing it (temporarily of course) and see if it resolves your memory leak issue(s). It fixed the issue on our server after removing the carrier group, although we narrowed it down to the Stennis itself, we removed the entire group for reasons of not wanting a group navigating the seas without the carrier. Sucks, but a temporary fix for now. Somewhat seems to be tied/associated with the new LSO audio/menu interaction that was introduced. Possibly was another thing that was not supposed to be released, but found it's way in...

 

 

Cheers,

 

Don

[uri_ba]

For us ot seems that the trigger of the problems was someone strapping to the catapult. AI or human.

 

When an AI tried, we all so him hover in the air and server hung.

After a restart, i tried to strap in (different mission differet theater.

 

It took me a few presses on the U key (even though i was lined up perfectly).

 

Once strapped, the other two saw me buried in the deck.

Tried to lauch and see if that frees it.. but mission did not accept my launch cue (i was in the 'cat).

[Ziptie]

For us ot seems that the trigger of the problems was someone strapping to the catapult. AI or human.

 

When an AI tried, we all so him hover in the air and server hung.

After a restart, i tried to strap in (different mission differet theater.

 

It took me a few presses on the U key (even though i was lined up perfectly).

 

Once strapped, the other two saw me buried in the deck.

Tried to lauch and see if that frees it.. but mission did not accept my launch cue (i was in the 'cat).

 

Yes, it's the carrier causing issues. We had someone hook up to the cat, caused an issue. We had someone taxing to the cat and drop their launch bar, caused an issue. We had someone enter into a hot aircraft by the island, as soon as they started to taxi, caused an issue. Something with it is causing Moose script failures as well. It's a shame. We removed the carrier from the server and voila, sorted. No more crazy memory leak on the server machine.

 

 

Cheers,

 

Don

[lbellouny]

We have been having the same issue with our dedicated server for our group.

 

We pared the mission back to the basest essentials, and the longest we were able to get the server to stay up was 2 hours.

 

It then, within 3 minutes, went from 54% memory usage of the 32GB of RAM on the server to 98%, and caused it to crash.

[hlfritz]

our group tried a couple different missions. the high memory usage (up to all free memory on the server) started to climb after the 3rd person joined the server with a stennis present mission. this makes it unusable when carriers are in missions.

 

 

we tried a mission with no carrier and all seems well.

 

 

we can corroborate the carrier-in-mission issue.

[PravusJSB]

Your error relates to an event handler, be aware that ED inserted new events into the list midway bumping all the proceeding event IDs, which may be the route of your issue. If you're looking for a specific event ID, either amend or change it to look for the string instead.

[Boomer_QLD]

I can also confirm this. We initially tested the server with two pilots and it was ok, but once more people joined for our weekly mission, the server went to 100% memory use within 5 minutes and we had aircraft frozen in the air and massive lag spikes. It was totally unplayable. The exact same mission running on a second 2.55 server was working flawlessly.

[hlfritz]

anyone know if the patch that was released a few hours ago fixes or helps with this issue?

[BIGNEWY]

please attach your server crash logs if you have them

 

thanks

[Osprey123]

anyone know if the patch that was released a few hours ago fixes or helps with this issue?

 

No, huge issues. :helpsmilie:

[hlfritz]

please attach your server crash logs if you have them

 

thanks

 

@BIGNEWY: what if the server does not crash? it only runs up all the memory available on the server (up to 11G on our server) and players cannot do much if anything (huge pauses - i.e. several seconds or more, cannot change roles, etc.). it also stops making log entries.

[BIGNEWY]

@BIGNEWY: what if the server does not crash? it only runs up all the memory available on the server (up to 11G on our server) and players cannot do much if anything (huge pauses - i.e. several seconds or more, cannot change roles, etc.). it also stops making log entries.

 

We understand and I do have a report in already, if you can attach your dcs log it will help.

 

thank you

[squinkys]

Hopefully this helps, it's the log from the Hoggit PGAW server. One of the other admins grabbed it for me, he says it's the only thing of interest from the log. We're still running 2.5.6 (sans Stennis) on our TNN server, if we have any other issues on that we'll update this post with more logs.

PGAW.zip

[Ziptie]

anyone know if the patch that was released a few hours ago fixes or helps with this issue?

 

We understand and I do have a report in already, if you can attach your dcs log it will help.

 

thank you

 

For what it's worth, "our" server was MUCH better today in terms of memory usage. Been up for 8+ hours and no issue at all with it. Appreciate the work that Eagle Dynamics is doing. Not sure how much was our server host / game master and how much was Eagle Dynamics - but clearly the patch/hot fix today in a solid step in the correct direction. The carrier is still an issue and requires them to still be removed from the server to stop issues from arising, but I am assuming that Eagle Dynamics is aware of this and working hard to address the situation.

 

 

Cheers,

 

Don

Edited February 22, 2020 by Ziptie

[hlfritz]

@BIGNEWEY:

 

OK, server log from yesterday attached. the relevant session starts on line 240; previous lines are from the server startup in case they are helpful and relevant. I cut a lot of lines in between as it was all a bunch of errors, almost (if not all) can't find proto. Those continue after players connect and the mission starts as well. There are a LOT of errors in the logs, but I do not know if this is normal or not - I have not had an occasion to review the logs like this before.

 

it all goes to hell in a hand basket when the 3rd person connects at line 2985. no further entries were entered into the log, memory usage shot up to 9+ GB (our missions usually run from 1.8GB to 2.5GB), players just stuck in game and not able to do anything. The server had to be killed.

 

We next ran a mission on NTTR and had 3 people with no issues. Let me know if you want that log as well for comparison.

 

Also, I am open to troubleshooting further if you all would like.

dcs.log.old.txt

[BoneDust]

After connecting to the CAT the server begins to get choppy until it eventually crashes within 2-4 mins. Also, other players see the person that launched sitting on the deck of the ship even if they are airborne. The other issue is after you land you are not able to launch again. After you connect to the CAT it will through you around and eventually off the CAT.

 

The DCS log file is very large and cannot open it in note pad. It is currently 2.4MB and growing as I was looking at the file in explorer.

 

I should add that after yesterday's update our server seems to be more stable and restarts faster. The carrier issue is a show stopper for us so a quick fix would be great.

 

Crash file.....

 

# -------------- 20200220-214403 --------------

DCS/2.5.6.43872 (x86_64; Windows NT 10.0.14393)

C:\Windows\System32\KERNELBASE.dll

# E06D7363 at C5254C48 00:00000000

SymInit: Symbol-SearchPath: '.;D:\Program Files\Eagle Dynamics\DCS World OpenBeta Server;D:\Program Files\Eagle Dynamics\DCS World OpenBeta Server\bin;C:\Windows;C:\Windows\system32;SRV*C:\websymbols*http://msdl.microsoft.com/download/symbols;', symOptions: 530, UserName: 'Administrator'

OS-Version: 10.0.14393 () 0x110-0x3

0x0000000000034C48 (KERNELBASE): (function-name not available) + 0x0

0x0000000000004890 (VCRUNTIME140): (function-name not available) + 0x0

0x0000000000122ED3 (edCore): (function-name not available) + 0x0

0x0000000000121CC1 (edCore): (function-name not available) + 0x0

0x00000000000CD71D (edCore): (function-name not available) + 0x0

0x0000000000100C25 (edCore): (function-name not available) + 0x0

0x0000000000100A08 (edCore): (function-name not available) + 0x0

0x000000000084226F (DCS): uiAsyncNet::onGameStop + 0xC0FF

0x0000000000846F7E (DCS): uiAsyncNet::onGameStop + 0x10E0E

0x0000000000841B52 (DCS): uiAsyncNet::onGameStop + 0xB9E2

0x00000000008690DF (DCS): uiAsyncNet::onGameStop + 0x32F6F

0x000000000086915F (DCS): uiAsyncNet::onGameStop + 0x32FEF

0x00000000003F9F3B (DCS): woLA::loadNewInputLayout + 0x21DDB

0x00000000003F36A3 (DCS): woLA::loadNewInputLayout + 0x1B543

0x0000000000103D60 (edCore): (function-name not available) + 0x0

0x00000000008563BF (DCS): uiAsyncNet::onGameStop + 0x2024F

0x0000000000856971 (DCS): uiAsyncNet::onGameStop + 0x20801

0x0000000000869C8B (DCS): uiAsyncNet::onGameStop + 0x33B1B

0x000000000084716E (DCS): uiAsyncNet::onGameStop + 0x10FFE

0x0000000000836FC2 (DCS): uiAsyncNet::onGameStop + 0xE52

0x00000000006ED334 (DCS): dDispatcher::InitDatabase + 0xE0A4

0x00000000006C3754 (DCS): woLA::loadNewInputLayout + 0x2EB5F4

0x00000000006C3B19 (DCS): woLA::loadNewInputLayout + 0x2EB9B9

0x0000000001743D46 (DCS): AmdPowerXpressRequestHighPerformance + 0xBC9D42

0x000000000089D54E (DCS): uiAsyncNet::onGameStop + 0x673DE

0x00000000000084D4 (KERNEL32): (function-name not available) + 0x0

0x000000000006E8B1 (ntdll): RtlUserThreadStart + 0x21

Edited February 21, 2020 by BoneDust

[PravusJSB]

BoneBust, I ge tthe same thing. I've removed Stennis and its fine again.

[wopski]

Your error relates to an event handler, be aware that ED inserted new events into the list midway bumping all the proceeding event IDs, which may be the route of your issue. If you're looking for a specific event ID, either amend or change it to look for the string instead.

 

Thanks dude!