Operation "Blue Flag" - 24/7 PvP Campaign - ROUND 5

[LevelPulse]

This was a very nice read Maverick, one thing which i wonder about. Does warping also get affected due to other programs running in the background? So let's so a program was using 13% (TrackIR) would this cause warping? Because essentially the server would use 9% + trackir?

 

I've never experienced such issues, unless it only occurs when players with bad ping join.

[104th_Maverick]

The %s I mentioned are only applicable to the host machine running DCS, not the clients machine.

[104th_Maverick]

So what you are actually talking is not "ping" but QoS and packet flows which are a totally different thing!

 

 

Sort of.

 

What Im trying to say is High Pings given by the DCS Player Table are indicative of poor QoS. Not in every single cases by but far in most cases in my experience, if a client has a ping of over 350 its because his connection to the server is poor.

 

When I monitor it in real time I see packet loss and heavy fluctuations in latency.

 

The exception here is people who geographically live far away from the server but can connect on solid networks. This is where my argument breaks down.

 

If a client has a very stable connection of say 300 and his ping is only fluctuating like everyone elses by 20-30ms per second then this causes no addition loading on the CPU.

 

However in our experience from hosting 50-60 players every day for the past few years when we monitor a clients connection who is getting out of control it causes DCS to start overloading our CPU, no matter how many upgrades we throw at it.

 

And time and time again when we kick or ban this client from the server we watch the CPU % gradually return back to normal and the clients in the server report that the lagging has stopped.

 

This cannot be a co-incidence ...

 

Like I mentioned earlier, its not a fix it all solution, but ping limits do increase server stability.

 

The 104th server is not bullet proof by any means, Im not here waving that flag. But there is no doubt to us at all, that DCS runs much more solidly with a ping limit of approx 330 than it does without one.

 

I have watched our CPU recover from high loads literately 100s and 100s of times after removing a high ping client from the server who has an unstable connection*

After every time I've seen this happen a member of the public connected to our teamspeak has confirmed to me that the warping on the server has stopped, now that the client in question has been removed.

 

*disclaimer - Im talking about High Ping clients who's real time connections speeds are fluctuating by factors of 100s not 10s

[sirscorpion]

PD: I don't want to start a flame war I Want to understand how it works

 

No worries, I am not IT expert, but I ran server for long time, from wiki

 

"most common type of lag is caused by network performance problems. Losses, corruption or jitter (an outdated packet is in effect a loss) may all cause problems, but these problems are relatively rare in network with sufficient bandwidth and no or little congestion. Instead, the latency involved in transmitting data between clients and server play a significant role. Latency varies depending on a number of factors, such as the physical distance between the end-systems, as a longer distance means additional transmission length and routing required and therefore higher latency. Routing over the Internet may be extremely indirect, resulting in far more transmission length (and consequential latency) than a direct route, although the cloud gaming service OnLive has developed a solution to this issue by establishing peering relationships with multiple Tier 1 network Internet Service Providers and choosing an optimal route between server and user.[2] In addition, insufficient bandwidth and congestion, even if not severe enough to cause losses, may cause additional delays regardless of distance. As with the hardware issues, packets that arrive slowly or not at all will make both the client and server unable to update the game state in a timely manner."

 

So basically there are several factors that effect game performance, each one is diffrent, each one has a different effect "even though they might look similar"

 

-Too many players = Network congestion, This holds hands with ping as well, so basically the server does not have enough bandwidth to send out information to every one in the "game/ cycle window". players with lower ping will get the information first, as they can download that info faster.

 

-Too many players might also cause CPU use, but that has alot to do with scripts.

 

-Player with High latency and server having no lag compensation. This can happen in 2-3 ways.

1. Player With high ping is sending the info later than any one else, AKA late packet. Basically his input effects no one but him self for the immediate move, but will take effect for a higher move or a more important one "depending on network game architecture"

 

If the server has bad lag compensation

2. Player with high ping is causing the server to wait for him to DL/UL his move, there for makes every one wait for it. so lets say you send information at 100 sec game time, i send it at 101, and the lager at 110. Basically the server will be like aright no moves for 10 sec till i figure this out. and then it works on the other way down so basically you see things move back or forward "warp"

 

Every game works like this in principle, Pure Client to Client means that the server will wait for every one to do the move before the next event or next move, not just players but even AI stuff and game objects "missiles bullets etc".

 

Simple way. i have 200 ping, i fire a missile, that move goes to server, server tells every one scorp fires a missile, renders it shows it on ur screen 200ms + your ping to the server 150ms, but also every other player so a 600ms player will see it 200+400. "but you a lower ping guy say it 350ms so your game tells the server i saw the missile" then the other player tells it 600ms later. and this one way between 3 players, now make that 60 and 2 way and you can see how even 300-400 ping is an insane amount for any game.

 

Now depending on the Network system DCS use there can be many results all of them will be bad, while the high ping player wont know what happens and that every thing is working fine. But for high ping it will be an effect known as desync, some games will slow time " EVE online, Wargame Red dragon", others with authorize the server to make the final call "warthunder, BF3". Some will compensate "to a point ping limit" IL2, ArmA. Seems that DCS has none of that, at least as far as i can see.

 

This just one very small aspect of netowrking and online lag, you have script errors, packet loss, Low cpu cycles, frame rate and a 100 more other things.

Edited February 24, 2016 by sirscorpion

[0xDEADBEEF]

You have any valid metrics or explanation from ED that backs your statement above?

 

Again I have no knowledge on ED network code but what you state from network perspective it does not sound right.

Actually based on your stated above everyone who has a lower ping should see things in "slow motion", and I mean everything even moving from point a to b.

 

 

PD: I don't want to start a flame war I Want to understand how it works

 

While I don't know the netcode itself, this info from maverick adds another piece to my puzzle of how it works. It obviously fails at predicting positions when there is packet loss but is still using quite a lot of cpu to calculate that warping. I've also noticed that putting a load on a server (like going to F10 and zoom around on a server, everything keeping the cpu busy) will cause warping on the clients. Also if the client alt-tabs, takes a screenshot or whatever, keeps dcs (the client) from sending packets. So whenever the server has to deal with packet loss it fails and seems to put load on the server, the more load the less good the server can handle everything and the more warping you see. I got no clue what kind of code they implemented that causes behaviour like this, but this is the behaviour I observed. The only thing the game would need to do in simple speaking is: "i got no packet, lets guess the next position the airplane might be and dont bother much more until the next packet".

I'm still missing pieces of the puzzle to understand the current algorythm, like does the server calculate physics at all, or is it all done client side (which would make prediction less easy, but reduce cpu-load on the server). Also, if it was as simple as I described it would be relatively easy to fix/the fact it wasnt fixed yet indicates its not that simple. In any case I would be surprised if these observations are very far from the truth.

[RIPTIDE]

Maverick this is very interesting & thanks for sharing!

 

So what you are actually talking is not "ping" but QoS and packet flows which are a totally different thing!

What you need to monitor is fluctuation on packet flow.

 

 

 

 

Side note: I can show the same behavior on the server after and hour on the session with most of the clients with and avg of 120

That would be ideal. Average Ping is only an indicator. It's guys jumping around to 1000ping+ all the time, while maintaining a <300 average.

 

Although in fairness, there is a correlation between high ping and jittery game experience. My ping to Tokyo is 290 right now. That's 10,000km away. Anyone with 300+ ping, has problems that are not solely related to distance.

 

Ping also is a very fluid thing, as guys with 1MB connections, start to see increase pings as their connection is red lined after a duration in a busy server.

[NaCH]

;2690621']Sort of.

 

What Im trying to say is High Pings given by the DCS Player Table are indicative of poor QoS. Not in every single cases by but far in most cases in my experience, if a client has a ping of over 350 its because his connection to the server is poor.

 

When I monitor it in real time I see packet loss and heavy fluctuations in latency.

 

The exception here is people who geographically live far away from the server but can connect on solid networks. This is where my argument breaks down.

 

If a client has a very stable connection of say 300 and his ping is only fluctuating like everyone elses by 20-30ms per second then this causes no addition loading on the CPU.

 

However in our experience from hosting 50-60 players every day for the past few years when we monitor a clients connection who is getting out of control it causes DCS to start overloading our CPU, no matter how many upgrades we throw at it.

 

And time and time again when we kick or ban this client from the server we watch the CPU % gradually return back to normal and the clients in the server report that the lagging has stopped.

 

This cannot be a co-incidence ...

 

Like I mentioned earlier, its not a fix it all solution, but ping limits do increase server stability.

 

The 104th server is not bullet proof by any means, Im not here waving that flag. But there is no doubt to us at all, that DCS runs much more solidly with a ping limit of approx 330 than it does without one.

 

I have watched our CPU recover from high loads literately 100s and 100s of times after removing a high ping client from the server who has an unstable connection*

After every time I've seen this happen a member of the public connected to our teamspeak has confirmed to me that the warping on the server has stopped, now that the client in question has been removed.

 

*disclaimer - Im talking about High Ping clients who's real time connections speeds are fluctuating by factors of 100s not 10s

 

 

Thanks for explaining this is and is great that you shared this knowledge.

 

I don't know if it make sense from the IT perspective but to me experience worth much more than any theory !

[NaCH]

While I don't know the netcode itself, this info from maverick adds another piece to my puzzle of how it works. It obviously fails at predicting positions when there is packet loss but is still using quite a lot of cpu to calculate that warping. I've also noticed that putting a load on a server (like going to F10 and zoom around on a server, everything keeping the cpu busy) will cause warping on the clients. Also if the client alt-tabs, takes a screenshot or whatever, keeps dcs (the client) from sending packets. So whenever the server has to deal with packet loss it fails and seems to put load on the server, the more load the less good the server can handle everything and the more warping you see. I got no clue what kind of code they implemented that causes behaviour like this, but this is the behaviour I observed. The only thing the game would need to do in simple speaking is: "i got no packet, lets guess the next position the airplane might be and dont bother much more until the next packet".

I'm still missing pieces of the puzzle to understand the current algorythm, like does the server calculate physics at all, or is it all done client side (which would make prediction less easy, but reduce cpu-load on the server). Also, if it was as simple as I described it would be relatively easy to fix/the fact it wasnt fixed yet indicates its not that simple. In any case I would be surprised if these observations are very far from the truth.

 

Dead I agree on what you said & I think Marverick / sirscorpion experience worth more than any theory.

 

Going back with a "normal netcode" packet loss "should" only affect the player that is having it and not make everyone else warp like crazy. Maybe hear is were DCS netcode fails to id and make the CPU to spike... why beats me.

What usually Network code does is when there are packet loss it predict the next position of the object base on the last know parameters OR flicker / warp until it receives a next valid packet.

 

Ping is latency. Basically I live in ARG, I will always have 300 of ping with EU server because that what i takes the packets to travel.

All our connections goes to US (180 ping) and from there we jump to EU. Now, I know the QoS of my provider is enterprise grade with a fiber connected to level 3 infrastructure

 

http://maps.level3.com/default/#.Vs5PNigrKUk

 

What this should cause is latency.. like I will se the missile on my screen 300 MS later that actually hit me and also the server will register my inputs 300 ms later and transmit that to everyone.

[*Rage*]

I'm in two minds about this lag issue since I want the events to be as inclusive as possible. Im tired of missiles and gun track solutions doing the crazy chicken though. Its definately not all the time but its pretty frequent.

 

On the other hand if the Blueflag team feel its due to scripts then it may be unavoidable since the scripts are what make Blueflag so amazing!

 

Congrats to the organisers again:)

Edited February 25, 2016 by ///Rage

[0xDEADBEEF]

On the other hand if the Blueflag team feel its due to scripts then it may be unavoidable since the scripts are what make Blueflag so amazing!

 

Even if we would assume the scripts are the cause, DCS should be easily able to handle it. It's no complicated maths and we have i7s and SSDs. Once the problems are resolved, the scripting engine will deliver us a lot of joy, made by the community. I see blueflag as a taste of the future of DCS.

Yes, I'm rather new and overly optimistic, I know. I like dreaming :)

 

Congrats to the organisers again:)

 

I can only second that!

[Sweep]

How many Bluefor does it take to capture a FARP?

[NaCH]

How many Bluefor does it take to capture a FARP?

 

Is not a bout quantity, is about what you need to do.

 

First you need to close the FARP and then drop troops to capture it.

 

But remember ... the FARP has engineers that will repair it if you don't drop troops.

Is all in the SOP.

[Sweep]

I needed that laugh.

 

Thank you :)

[xcom]

I ask a simple thing, give me proof that the warping is caused due to ping. You cannot show that this is the reason.

 

There can be a million other reasons and from what we saw, pings are not the reason. As you said mav, we have a function that is now not operational, it is off because we saw it did not matter.

 

I believe, and again this simply guessing on experience as I cannot prove it, that the warping issues are related to DCS Net code and it's inability to hold the many events occurring in blue flag.

 

Because we want to give any chance to stabilize blue flag, we will put the ping limiting on to see the results again.

[sirscorpion]

I ask a simple thing, give me proof that the warping is caused due to ping. You cannot show that this is the reason.

 

There can be a million other reasons and from what we saw, pings are not the reason. As you said mav, we have a function that is now not operational, it is off because we saw it did not matter.

 

I believe, and again this simply guessing on experience as I cannot prove it, that the warping issues are related to DCS Net code and it's inability to hold the many events occurring in blue flag.

 

Because we want to give any chance to stabilize blue flag, we will put the ping limiting on to see the results again.

 

From my understanding running arma Dedicated servers "which i think are no diffrent from DCS but with far more tools and control over the server, and some compensation", that events are handled by the CPU and game engine and not the net code. What is handled by the net code is event distribution, if those events are too big "large packets" or too many "bottleneck" then that is an issue.

 

However from my observation, that when this happens the server will drop frames, example a some one firing mini guns. this is directly effected by how many messages the server can send, "in ArmA i can set that per player, so in the end you get a total number Per player X number of player X number of AI"

 

Packets in any game come in terms of priority, Guaranteed, Non guaranteed, And error packets.

 

-Guaranteed are the most important ones, kill or hit or collision usually fall under this one, its usually simple and binary. Game usually will crash if some one misses a Guaranteed message.

 

-non-Guaranteed, those handle stuff like, location, ammo count etc, this is something that wont crash, but you will see artifactting.

 

-error packets, this tells a client that an error happend, and compinsates for it "if the engine can do it"

 

 

How to test for This if script vs ping:

 

1. to find out if its bottle necking: test if server is using more bandwidth in the first 1 hour Vs the last 1 Hour. If this happens then more information is being generated "equal number of players"

-Sub test:

 

-Reduce scripts needs, if fuel runs are 1k then make them 10k. If bandwidth use is the same then 1 thing eliminated.

 

- reduce number of bases and objectives half a map.

 

 

In regards to Ping. any online game has cycles. think of it like a game of roulette or monopoly or any table top game. every one gets a turn. every one plays their hand, and at the end of the turn server tells every one what happen. IF you have 1 player that's late in playing every one else will get what happen after he played his hand. This is the case with any Mplay game ever made and will be made for the distant future.

 

In other games the "late, lagging" player is compensated by :

 

A: server creates a fake player to hold the information then dumps the info on the lager at once "most FPS, Some sims", you see your self warping, your self insta killed "etc"

 

B: Server slows Time, "most RTS games"

 

C: Ping limit to match the cycle of the game, you cant play Any fps game with 150+ ping, RTS around 300-400"

 

I hope this does not frustrate you just trying to explain it, from my experience "note i have no idea how DCS does this technically but i know how many other games do it" you guys did an amazing job with this and i would like it to run as smooth as possible as well.

 

there are many tests that can be done, heck asking ED for how the network works is a simple one for understanding network compensation, all my findings have been from observations only

[NaCH]

I ask a simple thing, give me proof that the warping is caused due to ping. You cannot show that this is the reason.

 

There can be a million other reasons and from what we saw, pings are not the reason. As you said mav, we have a function that is now not operational, it is off because we saw it did not matter.

 

I believe, and again this simply guessing on experience as I cannot prove it, that the warping issues are related to DCS Net code and it's inability to hold the many events occurring in blue flag.

 

Because we want to give any chance to stabilize blue flag, we will put the ping limiting on to see the results again.

 

 

Please do, I don't want to be the cause of people having a bad experience.

As I said blue flag is the best thing that could happen to dcs and I'm ok ok to wait when the same mission is in another place.

 

Hey I waited 11 years to have somethings like a dinamic campaign and can wait more !

I only dream that Ed will dedicate all the efforts on improving the engine and mp and leave the modules to third parties

[NaCH]

From my understanding running arma Dedicated servers "which i think are no diffrent from DCS but with far more tools and control over the server, and some compensation", that events are handled by the CPU and game engine and not the net code. What is handled by the net code is event distribution, if those events are too big "large packets" or too many "bottleneck" then that is an issue.

 

However from my observation, that when this happens the server will drop frames, example a some one firing mini guns. this is directly effected by how many messages the server can send, "in ArmA i can set that per player, so in the end you get a total number Per player X number of player X number of AI"

 

Packets in any game come in terms of priority, Guaranteed, Non guaranteed, And error packets.

 

-Guaranteed are the most important ones, kill or hit or collision usually fall under this one, its usually simple and binary. Game usually will crash if some one misses a Guaranteed message.

 

-non-Guaranteed, those handle stuff like, location, ammo count etc, this is something that wont crash, but you will see artifactting.

 

-error packets, this tells a client that an error happend, and compinsates for it "if the engine can do it"

 

 

How to test for This if script vs ping:

 

1. to find out if its bottle necking: test if server is using more bandwidth in the first 1 hour Vs the last 1 Hour. If this happens then more information is being generated "equal number of players"

-Sub test:

 

-Reduce scripts needs, if fuel runs are 1k then make them 10k. If bandwidth use is the same then 1 thing eliminated.

 

- reduce number of bases and objectives half a map.

 

 

In regards to Ping. any online game has cycles. think of it like a game of roulette or monopoly or any table top game. every one gets a turn. every one plays their hand, and at the end of the turn server tells every one what happen. IF you have 1 player that's late in playing every one else will get what happen after he played his hand. This is the case with any Mplay game ever made and will be made for the distant future.

 

In other games the "late, lagging" player is compensated by :

 

A: server creates a fake player to hold the information then dumps the info on the lager at once "most FPS, Some sims", you see your self warping, your self insta killed "etc"

 

B: Server slows Time, "most RTS games"

 

C: Ping limit to match the cycle of the game, you cant play Any fps game with 150+ ping, RTS around 300-400"

 

I hope this does not frustrate you just trying to explain it, from my experience "note i have no idea how DCS does this technically but i know how many other games do it" you guys did an amazing job with this and i would like it to run as smooth as possible as well.

 

there are many tests that can be done, heck asking ED for how the network works is a simple one for understanding network compensation, all my findings have been from observations only

 

Arma rv engine is a totally different thing And I know and have vast experience on dedicated server.

 

Arma rv I think does not have anything to compare with dcs.

[sirscorpion]

Arma rv engine is a totally different thing And I know and have vast experience on dedicated server.

 

Arma rv I think does not have anything to compare with dcs.

 

Networking is all the same, game engines difference is simply how they compensate. this internet protocol. If DCS uses a different internet protocol and some how then color me impressed.

[QuiGon]

When I read all this about server stability and given the importance of BlueFlag and it's pioneering work regarding DCS MP gameplay, I would really like to hear some words from an ED developer about the current DCS netcode and it's future.

[Ciribob]

Please make sure you all add the correct autoexec.lua as detailed in the first post to increase the bandwidth that DCS can use: http://forums.eagle.ru/showpost.php?p=2674564&postcount=1

 

This should help with "lag" but it really only works best if ALL players on the server have it installed. :)

 

Info Duplicated here:

 

Place the autoexec.cfg file to your Saved Games/DCS/Config folder

 

Link: http://www.foinikas.org/ftp/public/DCS_World/Briefings/Buddy%20Spike/autoexec.rar

 

Contents of autoexec.lua for 8Mbit down and 2Mbit up:

if not net then net = {} end
net.download_speed = 1024*1024
net.upload_speed = 256*1024

Edited February 25, 2016 by Ciribob

[sirscorpion]

Please make sure you all add the correct autoexec.lua as detailed in the first post to increase the bandwidth that DCS can use: http://forums.eagle.ru/showpost.php?p=2674564&postcount=1

 

This should help with "lag" but it really only works best if ALL players on the server have it installed. :)

 

Info Duplicated here:

 

Place the autoexec.cfg file to your Saved Games/DCS/Config folder

 

Link: http://www.foinikas.org/ftp/public/DCS_World/Briefings/Buddy%20Spike/autoexec.rar

 

Contents of autoexec.lua for 8Mbit down and 2Mbit up:

if not net then net = {} end
net.download_speed = 1024*1024
net.upload_speed = 256*1024

 

I have done that personally, But without every one doing it it will be hard to observe difference.

 

Is there a possibility to get an official word from ED to how the net-code works? whats the good upper limit to ping and so on? should help a long way with trying to get stability.

 

some updates on the new net code or dedicated severs will be highly appreciated from the community as well.

[Pikey]

I'm going to wade in here and tell you all you are all wrong about the cause of the issues in the BF server because it's down to unit count and updates in the netcode. You can reproduce. Treat every non static unit as a unit and count them, especially if they are moving. This means all the missiles, crates and vehicles and infantry that spawn when ejecting and all that. The more you have, the worse the performance gets in multiplayer. You can test it in CPU % reliably. You can also see HUGE differences in servers like 104th where they don't go overboard on unit counts and BF. All units with sensors (visual included) create extra load as they need to check for other units in sensor range.

 

You have a blank 10v10 client mission with nothing else, she will run smooth. Have 30v30 it begins to degrade. Have 30v30 with spawning units everywhere, moving ground troops etc and unit counts in the 300+ and it goes to ratshit. It's simple scaling.

 

With regards to high ping players, you will see them warp when they have issues client side (thats an ED problem btw, think back to the alt-tab and screenshot days) But what you generally have is complaints about both sides warping and thats where the server side comes in. Its the DCS application, not the data centre, not neccessarily the mission, not neccessarily the individual ping or variable latency either (although each can contribute on top of the main root cause).

 

The answer that Greg and Xcom can deal with is two servers. They cannot really change the rest.

[NaCH]

When I read all this about server stability and given the importance of BlueFlag and it's pioneering work regarding DCS MP gameplay, I would really like to hear some words from an ED developer about the current DCS netcode and it's future.

 

Plus 100000000000000000000000 on this.

[HiJack]

We need dedicated server that can render 2000 units and 128 human players :D

[RIPTIDE]

I ask a simple thing, give me proof that the warping is caused due to ping. You cannot show that this is the reason.

 

There can be a million other reasons and from what we saw, pings are not the reason. As you said mav, we have a function that is now not operational, it is off because we saw it did not matter.

 

I believe, and again this simply guessing on experience as I cannot prove it, that the warping issues are related to DCS Net code and it's inability to hold the many events occurring in blue flag.

 

Because we want to give any chance to stabilize blue flag, we will put the ping limiting on to see the results again.

Is your CPU usage <100% (single core aggregate), but you still see server side FPS drop after time?