AWACS are uploading package from clients causing game freeze ?

[Tomy10]

Hi there !

 

With the squadron we are running missions on a dedicated server. On one mission, we have a very strange problem we struggle a lot to fix.

 

For each client the game microfreezes every 10-12 sec. The freeze is synchronized with all client.  the microfreeze is very constant, and happen whatever the graphic or the client setup.

 

We have identified a specific value thats changes very similarly to the freeze : on each microfreeze the client is uploading a package : the bigger the package, the bigger the freeze :

 

 

 

 

The problem is not showing on solo, and it does not happen to the host.  Its a client problem in link with the mission.

 

Our Dedicated server can't be the source of the problem : who ever host, the clients have the freeze : so its a client or mission issue

Server specs :

Z270-E

i7 7700k

16G 3000Mhz

SSD
1Gb on Ethernet cable.

--> it cant be a server hardware/software problem, on other missions it runs perfectly fine.

 

Client : It can't be the clients fault cause who ever it is, the problem happen. Im localy connected to the server. Some other are internet connected, some are wifi, other are cabled.

My client PC specs :

i7 10700k

16Gb 3000Mhz ( i know its not enough, 32 on there way :) )

RTX 3070

NVME SSD 4Gb/S

Wifi 400Mbts up/down

As a client, I have 2-4 ping on the server.

Every client have Reshade. Can't be the problem cause no freeze on other mission/server

-> it cant be a client pc hardware/software : whatever to specs  or  the mission  or on other server it runs perfectly fine.

 

So it must be a mission problem. The mission is Iran Chess https://www.digitalcombatsimulator.com/fr/files/3314295/

 

The " upload spike " only occures on this mission. But we can't figure WHY its happening. Sometimes, very very rarely, client have no spikes at all !

 

I've tried to : disable MOOSE, disable EVERY script. Only the value of the spike is changing : the more is on the mission, the bigger the upload spike. We dont feel the freeze if the spike is lower than 12kb~, we start feel the freeze from 20kb~ (sorry i dont know the units, its the number shown in dcs connexion box)

They're is no mod on the mission.

 

Clients and server have disable windows defender stuff etc...

-> so problem does not happen in solo

 

So we think that the problem is : something the mission ask the client to send. This thing is size is changing depending on what on the mission (no trigger 10kb/s, with every trigger : from 20 to 50 kb/s )

 

 They are errors is the clients and the server logs. But i dont know where to start... the very specific thing is the interval beetwen the spikes/microfreeze: its constant whatevere the client is doing: on ground, on air, what ever the aircraft he flyies, his computer specs etc...

 

Actually i trying to figure out : whats happening every ~10sec in the mission ?

-its not realtaed to Moose : no diffrence with/without

-no related to trigger : with all or none : no difference

- client logs says ZSU57 and Ural skin cant be find : removed ZSU57 and change Ural skin

HELP ! XD

dcs_server.log

dcs_client.log

Edited February 23, 2021 by Tomy10

[Tomy10]

Update : after many troubleshooting, i've found the origine of the problem, and i cant believe it... Its the blue AWACS !!!!! There is 2 Red Awacs and 2 Blue Awacs. I've removed the red awacs : no changes. I've remove one blue awacs : package reduced by 50%. I've removed the 2 blue awacs : NO PACKAGE UPLOADED ! 

 

I'm going to invastigate a bit more.

Edited February 23, 2021 by Tomy10

[Tomy10]

Well good news is : i'm now absolutely convince that the package/freeze were coming from the blue Awacs. I've tried to place new blue awacs : E3/E2, différent altitude/speed etc, but the problem is happening everytime. No problem at all for red awacs.

 

Bad news is : Blue coalition AWACS are causing Freeze on my mission !!

 

Here, you can clearly see that there is no more upload spike. And no more game microfreeze every 10sec~.

 

[BIGNEWY]

We will take a look, do you happen to have a Miz or track replay we can look at showing it happen?

[Tomy10]

3 minutes ago, BIGNEWY said:

We will take a look, do you happen to have a Miz or track replay we can look at showing it happen?

 

This mission causing clients  to upload package every 10sec~ and microfreeze.

IRANCHESSV2.1_FREEFLY_ZEUS

IRANCHESSV2.1_FREEFLY_ZEUS.miz
 

13 minutes ago, BIGNEWY said:

We will take a look, do you happen to have a Miz or track replay we can look at showing it happen?

This is the exact same mission but withouth the Blue Awacs

IRANCHESSV2.1_FREEFLY_ZEUS_NOBLUEAWACS

IRANCHESSV2.1_FREEFLY_ZEUS_NOBLUEAWACS.miz

15 minutes ago, BIGNEWY said:

We will take a look, do you happen to have a Miz or track replay we can look at showing it happen?

 

The problem seem to happenning with :

 

BLUE AWACS, US, E3 and E2. Maybe more...

No uploading/freeze with BLUE TASK FORCE, KJ2000

[BIGNEWY]

thanks we will take a look 

[Tomy10]

Me again sorry haha

 

I've tried

 

BLUE TASK FORCE, E3 : no freeze

 

Might come from US coalition only ??

 

Well see you next time thanks for replying

[Tomy10]

We tryied the mission with new blue task force awacs. Everything was running perfectly. But suddently the upload/freeze started again -_-

 

If relaunch the mission : no freeze. Now it seems the freeze are not present at mission start, but could be triggered during the mission

 

we are investigating again.

 

EDIT :

 

With KJ2000, mission is running absolutely fine for more than 3 hours with 3 clients.

 

So Blue US E3/E2 might be the problem.

Edited February 23, 2021 by Tomy10

[BIGNEWY]

We are having trouble reproducing at the moment sorry. All seems fine when we run the missions, we will continue to monitor. 

 

thanks

[Knock-Knock]

Could it be Datalink / ELRS?

[Tomy10]

36 minutes ago, Knock-Knock said:

Could it be Datalink / ELRS?

Yes maybe, i did not try to remove that.

 

KJ2000 doesnt have ELRS. im going to try

[Tomy10]

1 hour ago, BIGNEWY said:

We are having trouble reproducing at the moment sorry. All seems fine when we run the missions, we will continue to monitor. 

 

thanks

 

1 hour ago, Knock-Knock said:

Could it be Datalink / ELRS?

Well...

 

there is a high chance that the problem comes precesly from EPLRS :

 

EPLRS ON on E3 :

 

 

 

 

 

EPLRS OFF on E3 :

 

 

EPLRS on :

 

IRANCHESSV2.1_FREEFLY_ZEUS_E3_EPLRSON.miz

 

EPLRS off :

IRANCHESSV2.1_FREEFLY_ZEUS_E3_EPLRSOFF.miz

Edited February 24, 2021 by Tomy10

[Tomy10]

On 2/24/2021 at 11:03 AM, BIGNEWY said:

We are having trouble reproducing at the moment sorry. All seems fine when we run the missions, we will continue to monitor. 

 

thanks

Hi again.

 

There is no doubt the stuttering is caused by EPLRS from AWACS. We fly more than 3 hours without EPLRS and absolutely no freeze. As soon as I turn it on they come back... Its a major issu for a BVR theatre...

[BIGNEWY]

We have seen it on your mission, when we try in a fresh mission it seems ok. We are still looking into it. 

 

thanks

[Tomy10]

14 hours ago, BIGNEWY said:

We have seen it on your mission, when we try in a fresh mission it seems ok. We are still looking into it. 

 

thanks

Ok. I might have done something really really silly do create this problem haha XD

 

I've investigate a bit more :

 

When EPLRS on AWACS is on, it makes the client tu upload a package. No EPLRS= no package. To reduce this package, reduce the number of awacs. I've did that to temporaly fix the mission.

 

I then tryied : what if i Had A LOTS of awacs, just to be certain this problem comes from EPLRS on AWACS :

 

the 'upload package' goes up to 15kb/s with 1 AWACS. So i copy/paste a lots of them : it goes up to 400 kb/s, freezing the game so bad XD

 

Task really shows the impact here :

 

See you later o7

[BIGNEWY]

please attach your test mission that reproduces this I will have the team look at it. thanks

[Tomy10]

1 hour ago, BIGNEWY said:

please attach your test mission that reproduces this I will have the team look at it. thanks

Just to remind that it happens only to clients, not to the host. We've tryied on different server too.

 

 

Mission with 2xAWACS, EPLRS OFF : No prolem

IRANCHESSV2.2_NOZEUS_EPLRSOFF.miz

Mission with 2xAWACS, EPLRS on : Package send every 10 sec bla bla bla , upload spike to 20kb~

IRANCHESSV2.2_NOZEUS_2xEPLRSON.miz

 

Mission wiith 20xAWACS, EPLRS on : blabla +200bk/s

IRANCHESSV2.2_NOZEUS_20xEPLRSON.miz

 

 

I've tried to remove every EPLRS on the theatre exept for 1 AWACS : no changes.

 

I've tried to remove the detection zone of Moose which related to AWACS,  in  the IRAN CHESS BLUE v2.2 .lua  : no changes

Edited February 27, 2021 by Tomy10

[GatorNutz]

On 2/27/2021 at 9:58 AM, BIGNEWY said:

please attach your test mission that reproduces this I will have the team look at it. thanks

I am an admin for the Growling Sidewinder servers and our clients are seeing the same spike.  We have been trying to diagnose serious lag issues that started 3-4 weeks ago.

 

We also have E-3 with EPLRS turned on.

 

For the time being, we have removed all ground and static units from the mission and the lag has improved significantly.  However, I wonder if this EPLRS issue is also part of the issue.