Multiplayer server freezes and crashes multiple times per day

[Samoht]

I run and maintain the Aerobatics Online servers. Our 1.5.8 release server has been freezing multiple times per day. This problem has existed since 1.5.8 was initially released on Open Beta and was brought over to Release with the last update. I managed to finally catch it in action and manually completed capturing the logs. Normally it just hangs and my automation scripts kill the process before it finishes getting/sending logs (because they don't send automatically, and they should).

 

Symptoms:

After some random period of time (could be 20 minutes, could be 120 minutes) connected players will begin to warp around. On the server's console side, the interface gets intermittent freezes (blue busy mouse indicator) or periodically switches to the black DCS logo screen. If I keep trying to interact with the console, Windows will eventually report that the DCS process is not responding. Eventually, my watchdog timer will fire noting that the simulation frames have stopped and kill the DCS process. If this is not done, the DCS process will hang forever (or until manual intervention). Players are usually not disconnected when this happens, but they are frozen in the sky.

 

There does not seem to be any particular trigger as to what causes this. The attached logs were when the server only had three people on, but it was after crashing once previously. Most times it crashes with 15+ players connected.

 

AO Caucasus is running a stripped down version of the mission with basic assets only. The only AI units are tankers. Ships are statically placed at runtime using external scripts. Other administrative functionality is handled through external scripts that are not included in the mission. I have tried running the server with the scripting interface down and still get similar performance/stability issues.

 

Server specs:

Windows 10 version 1709, i7 7600 (stock), 16GB ram, 1050ti 4gb video card (stock), SSD with 42gb free. This is a computer dedicated to running DCS World as a server. The computer automatically reboots after every game session (4 hours, currently) and does have a sound device enabled.

 

I should note that the DCS World multiplayer server hardware requirements have dramatically increased with this latest version. More than ever, the video card makes a difference whereas it just needed to support DirectX 11 before and a GT 720 would happily run a multiplayer server. That's no longer the case, even with options.graphics.render3D = false in the autoexec.cfg.

 

These stability issues are killing our player base. On a Saturday at mid-day in the US, especially right before Christmas, I should have 45 people online right now. Instead, I have eight.

dcs.log-20171223-203347.zip