Aerobatics Online Server Crashing Frequently

[Samoht]

We've been getting multiple crashes and UI freezes per day on the Aerobatics Online Caucasus server ever since the latest OpenBeta update. As other admins have pointed out, 2.5.0 was not doing this, but 2.5.1 is crashing a lot.

 

Unfortunately, we don't get any crash logs when the entire process freezes; however, an outright crash will generate a dump file. A dump file is attached.

 

We're powering through keeping and OpenBeta server up to hopefully provide some good data you can use for making the next patch better. In the mean time, I spun up a second AO Caucasus server on DCS World release. They're running pretty much the same mission, and the computers hosting have the exact same specs. It might make for a good comparison.

 

As a side note, aircraft seem to "hop" now when flying in close formation. It's not always apparent when observing someone far away, but in close proximity (less than six feet) the judder makes things nearly unplayable. It's not the world, just the other aircraft. We did not see this when flying on a small mission and it's not always present on AO either.

 

I also have not seen any significant reduction with mission load-in time when using a large preload radius, which is necessary to cache models and avoid spawn related lag spikes.

dcs.log-20180428-033349.zip

[Basilem84]

Thanks Samoth. Very difficult to play online right crash happen almost every 20-30 min today! Is it true that the Harrier is a trouble maker online ?

[Johnny_Rico]

The same thing is happening on our server UI freezes and its game over

[humptydumpty]

Hi Samo,

 

Since 2.5.1 for me it's better, AO seems to be running well for me. What i felt when your server resets on it's scheduled time it does it multiple times in a few minutes.

 

Did notice freezing of the other users and then it comes back to normal.

[sslechta]

Thanks Samoht. Here's the post I first reported 2.5.1 issues on with the AO server in case others would like to research...... https://forums.eagle.ru/showthread.php?t=207139

[Johnny_Rico]

how is the crashing going for you guys ?

[Samoht]

Still having on average 9 crashes per day. When it's running, it's pretty unplayable (for formation flying). It's been a long time since the server has been this empty for this long.

 

I'm continuing to reduce mission assets, but there doesn't seem to be any significant change with stability.

[Johnny_Rico]

is a new player spawning still the last message before server died?

Edited May 4, 2018 by Johnny_Rico
1.1

[sslechta]

Hey Samoht, I realize the AO mission is pretty complex but after some updates to Beta you can get better performance from building out the mission from scratch. Sometimes new changes just don't work well with missions created on previous versions. We had to do this several times in the past back in the VA days. It was a lot of work but the mission would start running better when constructed on the newest version of DCS.

[tovivan]

When (or if ever) DCS is stabilized and optimised back to levels of 1.5.8 (I'm quickly losing hope for that though), can I request that AO missions get one airfield with challenging conditions like Mozdok was in the VA days? Under that I mean rain, poor visibility, heavy crosswinds and so on?

 

Right now flying there is a bit... boring. It's perfect weather everywhere, daylight all day and unless I want to do the racing I get bored within some 10 minutes of joining (if I don't get disconnected or ctd-ed that is). Some challenge would be very welcome.

[Samoht]

Rico: Surprisingly, we're not really seeing the spawn error that you've been getting. If it has happened, it's infrequent enough that I don't notice. I've pulled the Harrier from our mission and doing so has drastically helped with warping issues. I've done several long flights when the server is full and not had any warps. Crashes are still happening though. Mainly on initial start. The DCS process crashes and then hangs when first loading. My watchdog script will get it eventually, but it delays server availability after scheduled restarts.

 

Slick: I rebuild the AO mission from scratch about once a year, but I haven't since 2.5 came out. With every new update, at minimum I save the mission in the new version and fix the airport inventories, but redoing Caucasus from the ground up is about three days of work. I should probably do it again for 2.5, but I'll wait for this current version to migrate to stable before undertaking that.

 

Tovivan: The mission editor is extremely unfriendly with customizing weather and getting it where you want it. It's pretty much random chance that you get something that works. In our case, I need to make sure that the winds are low enough at spawn airports but challenging enough at other distant airports. When we have racing going on, I also don't want the moutain courses to be covered in clouds. It's almost impossible to get all the weather to cooperate in all locations that matter.

 

That being said, we need to come up with some new activities and challenges. Doing more with the weather is high on my list. This is assuming that the weather bugs that got introduced with 2.2 have been sorted out. I haven't checked.

[Johnny_Rico]

Question: Do you try and prevent people connecting to the server until the MP mission has fully loaded? (Slot selection list is on screen)

 

Because I used to get crashes when people tried to connect while the server was loading up the mission, until that is I wrote a script to stop that happening

 

Oh and we had to update our mission, due to 2.5 and the parking slots changing

[Samoht]

Yeah, I noticed the parking spaces all adjusted. I spotted that one in the first 2.5.0 OpenBeta and got that sorted out.

 

We don't guard against people connecting on load, but then again, it's never been an issue for us (I don't know why, since it obviously is a problem for others). The crashes we're getting on load don't seem to be related at all to people connecting. We're seeing this:

 

2018-05-08 02:08:13.169 INFO EDCORE: C:\Program Files\Eagle Dynamics\DCS World\bin\World.dll

2018-05-08 02:08:13.169 INFO EDCORE: # C0000005 ACCESS_VIOLATION at 5A120700 00:00000000

2018-05-08 02:08:13.172 INFO EDCORE: 00000000 00000000 0000:00000000

2018-05-08 02:08:13.173 INFO EDCORE: 5A120700 6716DF10 0000:00000000 C:\Program Files\Eagle Dynamics\DCS World\bin\World.dll

2018-05-08 02:08:13.174 INFO EDCORE: 58F50848 6716E1E0 0000:00000000 C:\Program Files\Eagle Dynamics\DCS World\bin\AsyncNet.dll

 

It'll rarely load clean without a crash. Usually it's once. Sometimes it's twice. The process becomes non-responsive and the external watchdog will kill DCS and restart it after a few minutes.

[Johnny_Rico]

i just got another crash (mission running for a while)

 

Problem signature:

Problem Event Name: APPCRASH

Application Name: DCS.exe

Application Version: 2.5.1.16543

Application Timestamp: 5adf71a0

Fault Module Name: DCS.exe

Fault Module Version: 2.5.1.16543

Fault Module Timestamp: 5adf71a0

Exception Code: c0000005

Exception Offset: 00000000006df0a7

OS Version: 6.1.7601.2.1.0.272.7

Locale ID: 2057

Additional Information 1: b14b

Additional Information 2: b14bc36e280d5dc57b8831aa747e8580

Additional Information 3: 9adb

Additional Information 4: 9adbb216243e7cb755dca8669167b844

[Johnny_Rico]

another crash

 

2018-05-17 22:30:49.256 WARNING LOG: 1 duplicate message(s) skipped.

2018-05-17 22:30:49.256 INFO NET: client[2] is ready to start

2018-05-17 22:30:49.256 INFO NET: spawning client[2]

 

on spawn a client again