LOCERF 10-2 Announcement

[104th_Crunch]

Thanks Fudd for the efforts. Eventually I am sure we'll get there. We well have to see what works and what does not in terms of stability for an event. We can use 24/7 servers as an example to what to look for. Maybe DCs:BS might prove a more stable host over FC2. Need to test.

 

I hope that ED can continue to improve upon and focus on network efficiency for DCS. I hope that in the future multiplayer is the main focus of ED. I am so very thankful for this sim and hope things will only get better.

[Fahhh]

Sad to hear you had problems with the event. After all, did you eventually get to fly or did you postponed the event for a later date?

[Frostie]

I think the logical path to take with FC2 multiplayer large scale would be to run Lockon default, this means having no additional tools running in the background eg. ATC, Tacview, LEAVU etc. This would also be with 'exports off'. Surely with all these tools and 'export on' the information in such a large scale event is huge.

 

A lot a of time and effort are put into preparing these LOCERF's, thanks especially to Fudd for taking his time to create the mission and the mods and planners for putting so much in. So I think in the best interest of everyone we just walk before we start running and take this one step at a time.

[SEFalcon]

I think the logical path to take with FC2 multiplayer large scale would be to run Lockon default, this means having no additional tools running in the background eg. ATC, Tacview, LEAVU etc. ...

 

...and default track recording. I think current track recording option (fc 2.0) is BIG problem for resources. Totaly unnecessarily.

 

Sory to hear about problems on RF and i hope so you will fly it soon guys.

[MoGas]

...and default track recording. I think current track recording option (fc 2.0) is BIG problem for resources. Totaly unnecessarily.

 

Sory to hear about problems on RF and i hope so you will fly it soon guys.

 

That is right plus Tacview, you can see it on our Moonshield map that if you turn it down the Tacview, you have less stutter.

[Breakshot]

Failed on 4 different servers and under different configurations. FC2 (probably) couldn´t handle it. (crash, freeze, lag, etc etc)

 

Well technically it didnt fail on the second attempt on our server. But the fact that BLUEFOR had to have their precious datalink or whatever else they had (because we had IC on as we didnt have time to adjust it), while REDFOR had no qualms in sacrificing ATC capability in order to make this thing ACTUALLY work, was a bummer for me... sometimes you just have to sacrifice certain things, and certain BLUE individuals just boycotted instead.... :doh:

 

^^ @Frostie & Mogas, i agree completely, scripts crashed this event, IMO... we have seen this already in FC1 events before (take maple flag for example).

 

I think its safe to say we should try the next one with STOCK settings and then go from there... If that runs OK, then we can think of re-introducing additional tools...

 

Just my two cents...

[MoGas]

@Breakshot, only a handfull BLUE had LEVU runing, the hole 104th flew with ERI only and one pilot even without ERI.

 

But yes, for the next event who ever it will host, it would be needed to run it without any scripts too see what is going on, if the scripts making issues or something else.

[X-man]

But yes, for the next event who ever it will host, it would be needed to run it without any scripts too see what is going on, if the scripts making issues or something else.

Indeed, we dont know whats the cause at this point so every potential bottleneck should be tested.

[RvEFuSiOn]

FC2 doesn't seem to be very stable in handling events. I hope we can have some sort of test event every month at the very least once so that we can test the parameters of the game and find out what works and what doesn't

 

My request to ED is could you please work on stability for MP and how events are managed properly please for the next patch, many of us will be happy to contribute our time and machines to stress testing anything new.

 

@51'st guys it wasn't ERI or Leavu, yoda explained this very conclusively even on comms after the event, it was other factors. There were only 4-5 people max on Datalink and it doesn't use any extra resources from the server so this point is invalid, my guess is that it was the moving units, since they have to update their location's all the time to the entire server constantly, and couple that with the fact we had people connecting from LAN parties with weaker connection speeds.

 

 

We need to Figure out Connection speeds and what speeds each client should connect at when joining a server depending on his/her location and the Server's location. A speed test should be done prior to an event like this, so you know what speeds to connect at. This is how we did it in the Falcon4 Mp community and it helped tremendously and alleviated problems like this.

Example... If a certain client connects at too high a speed he could be hogging all the bandwidth of the server while others who have connected at lower speeds get the short end of the stick, and this is where a lot of the warping and lag also comes in.

 

We have lots of talented folks in this community and I'm wondering if someone can make some sort of mod or patch on how events are handled and if it can be done more efficiently.

 

 

Peace

Edited June 20, 2010 by =RvE=FuSiOn

[X-man]

@51'st guys it wasn't ERI or Leavu, yoda explained this very conclusively even on comms after the event, it was other factors.

We're not targeting ERI or LEAVU, but all of the exports (tacview, LOVP...). My question to you or Yoda, has this been tested on 50+ players or is it just in theory? Remember LRM which was also not suppose to give any effects, yet when put into events like this, things went to ****.

 

We should test it and only then rule it out. Not just say it's not that.

[RvEYoda]

We're not targeting ERI or LEAVU, but all of the exports (tacview, LOVP...). My question to you or Yoda, has this been tested on 50+ players or is it just in theory? Remember LRM which was also not suppose to give any effects, yet when put into events like this, things went to ****.

 

We should test it and only then rule it out. Not just say it's not that.

 

I don't recall what you refer to as "things went to ****" (unless you refer to the very old

coroutine based implementation), but that is not really relevant is it?

 

What is relevant is:

 

1. For the next event I recommend the following: Make a pack, a locerf-pack

for modman, where everyone gets the standard files. Everyone needs to be integrity

checked, not for this or that mod, but to ensure everyone has the same flight models,

weapon data etc, so that game does not desync and crash.

 

2. The server MUST run a stock setup. Zero mods or changes, nada, zip, null.

 

3. If you are going to host ATC, let a secondary host run it. do NOT risk the game host.

 

 

LRM was server side. Eri and Leavu are exclusively client side. They cannot

affect the server. Furthermore the (once again purely CLIENT SIDE) exports

of eri and leavu have far less cpu requirements than lrm ever had. Calling

local client exports the culprit is just as intelligent as saying "oh my mouse

driver is wrong so that crashed the game server"... or "my joystick has macros

so that eats more CPU cycles and crashes the server machine"

 

Furthermore both leavu and ERI have (unlike lrm) already been tested by a huge

number of people, and, it is built on the lessons learned from lrm.

 

One serious point people seem to be missing is that _you cannot disable export.lua from running:

- You can make them standard (not send data to external apps etc) but the export.lua is always running.

It always runs "LuaExportBeforeNextFrame" functions etc, whether you like it or not, or whether you

set anything to false or not, the variables are already exposed whether you intend to use them or not.

 

BUT!!

 

There is one possible crash issue with exports, but naturally this has to do with GLOBAL exports on SERVER.

Unlike Fc1, Fc2's function "LoGetWorldObjects" exports everything from aircraft down to every single bullet. The sheer cpu/memory load

of exporting all that data does take a LOT of fps, so that is why I really recommend that the game server

absolutely must run standard/stock/unmodded settings for large events. The game is a 32bit app and has

issues with running out of memory space since no cleanup is implemented (if it loaded one AC once, it stays

loaded), as discussed in other threads and confirmed by c0ff:

Therefor server must not run anything that increases memory usage. Stock setup, low gfx only!

 

In general people need to stop jumping to quick conclusions and instead start thinking logically, actually trying to work out

why something causes a crash. People need to stop using crash events like this as an excuse to bash other

software that has absolutely nothing to do with it. It will only accomplish 2 things:

 

1. Not get the crashes fixed

2. Create personal conflicts.

 

Do you want a long term deterministic solution or a "quick fix" that only out of sheer luck and coincidence happens

to work (due to magic of probability/statistics) the next time?

Edited June 20, 2010 by =RvE=Yoda

[Frostie]

I can understand the frustration felt after saturdays debacle, not least the fact that the option to run this mission using default options was put forward but not agreed as acceptable by the 'mod using crew' and an offer of a 'mod allowing server' pushed forward.

Its alright criticising people for being miffed at others actions but then there is no smoke without fire.

 

In hindsight it would have been a lot better for the whole event and everybodys time if after choosing the second server everyone switched to a default setting and flew LOCERF stock instead of like what happened of offering a server to allow the must have mods get used and end up getting nowhere. As it is now we don't even know if LOCERF will even run on default Lockon.

For the unsure and easily offended, stock means no tacview etc.

[RvEFuSiOn]

I agree about Tacview Frostie, this app seems to drop FPS for everyone quite substantially including myself.

I think Yoda is also right also about having some sort of LOCERF Modpack which would really improve things for everyone across the board. That way we would be on the same page and no opportunity for ambiguity.

 

Just curious guys, what mods were people using that could affect the server?

 

Peace

[RvEFuSiOn]

I think also the network code is not very efficiently managed

[104th_Crunch]

I suggest that we sacrifice mission secrecy, where participants cannot know position of enemy units and instead test the mission before hand. Realistically, many enemy positions would be known anyway because of satellite, UAV etc.

 

This way you stress test the mission before hand. Yes, there will few or no AI surprises, but the surprises can come from planning instead.

[SFJackBauer]

I would like first to say congrats to all squad leaders, and specially Fudd for putting this event forward. Even though we haven't flown properly, as a first-timer I learned a lot just by observing the preparations and planning.

 

I would like to suggest a scenario for testing:

 

- Prepare several missions (or even use an existing one as a template):

 

1) 50 player-controlled aircraft FC2 only

2) same as 1, but adding 10 KA-50 slots in the mix

3) same as 2, but adding small groups of ground units

4) same as 3, but adding large groups of ground units

5) same as 4, but adding triggers

 

- Run each scenario with 60 people (hardest part perhaps?) for 30 min, and check stability. Important to say that those people should be representatives from all over the world, to see if latency / speed / geography influences.

 

- If all runs well, then create other scenarios by adding other variables in the mix (LOTATC, tacview etc) one at a time.

 

Given that in 2 hours, 2 scenarios could be tested (given that people need time to connect etc), this could be done in a space of 4-5 weeks.

[RvEYoda]

Id like to propose something,, different. A kind of community wide stress test to see what the game can actually handle.

 

This would sound a bit extreme, but I propose all squads for 1 week to "lock" their servers

and host 1 common 200 slot server, to see how far you can take it before things go really

bad. Of course we would need a stable platform to begin with, and the key to this test is

to integrity check everything and enforce stock settings. This would be more of a

software test than anything else.

[Moa]

As good as TacView is, it does produce a lot of data that needs to be streamed to disk. This can induce stutter in clients. TacView should not be running on a server as the disk I/O can result in network lag (the PC architecture has never been great for this). Disk write activity without write caching can be a major source of lag. The downside of write caching is that data (such as your local mission track) is lost in the event of a crash.

 

Incidentally, I've been mucking around with LEAVU2 and it streams a few hundred bytes a dozen times a second between your (local) LockOn instance and LEAVU2 (mostly running on the same machine, but could be on your LAN). The network usage here is 2 kB/s tops. If it is on your machine then the time taken is 2 kB over your multi-gigabit bus speed. Depending on the driver details, zero-copy shared memory could be used. Even if you are streaming the data to another box on your LAN (as I do) this is 2 kb/s over a Gigabit link. The bandwith used is 0.002 % of capacity (0.02% if you have a 100 Mb/s connection). In short, the local LEAVU2 traffic is negligible and does not contribute to client lag in any way. It doesn't contribute to server lag either since the data for LEAVU2 is sourced from data you local LockOn client already has. I hope those numbers put the LEAVU2/ERI issue to rest with respect to alleged lag - use science not superstition please.

[104th_Crunch]

I know I mentioned all this in TS as well, but the 104th has stress tested with 50+ players. On more than one occasion. While we have problems, they are usually after a few hours, and not minutes like RF was this weekend. The problems we get after a few hours I believe to be memory management related. As Yoda said, things load into memory and stay in memory.

 

Since we tried the RF mission on the 104th server this makes me think, what was different from normal 104th server day to day operations?

 

- Different mission

- Hi group of players in China with high ping

 

That is all I can think of. Mods allowed in RF were the same as 104th server would allow a month or so ago. When I look at the RF mission, it is not really very large at all. Unit count looks to be not much different from 104th missions and triggers even less. A small number of units are moving however, in the RF mission. No units move in the 104th missions expect the last trigger in Moonshield 2. So we have not tested moving unit effect in FC2 much. We all know that moving units and a large number of players in FC1 would cause problems. Even if RF 10-2 mission is smaller than the 104th missions, it may be some design of the mission causing a bug issue.

 

Stress testing is the way to go. Like I mentioned previously, maybe forget about secrecy and stress test the mission that will be used in the event.

Edited June 21, 2010 by Crunch

[X-man]

Id like to propose something,, different. A kind of community wide stress test to see what the game can actually handle.

 

This would sound a bit extreme, but I propose all squads for 1 week to "lock" their servers

and host 1 common 200 slot server, to see how far you can take it before things go really

bad. Of course we would need a stable platform to begin with, and the key to this test is

to integrity check everything and enforce stock settings. This would be more of a

software test than anything else.

 

I also think this is the way to go. And as SFJackBauer suggested, we can introduce different parameters to gradually push the server...

[4c Hajduk Veljko]

We at =4c= are conducting a community wide event, semi dynamic campaign known as a NUCLEUS. We always conducted stress tests a week before the main event. The mission for the stress test was overloaded and with way more units then the actual event mission. Only after successfully passing the stress test we committed to the actual event.

 

Our NUCLEUS team has at least three members where, the first is the leader creating the mission and overseeing the whole process, the second is responsible for hardware operations and settings and the third is communicating the messages between participants.

 

I would like to thank Fudd and everybody else who was involved in RF preparation. I also would support Yoda's suggestion about shutting down all servers, maybe once a week for several hours and force everybody to go on one server to see what happens. This is something that can be negotiated, discussed and implemented through the Commanders Round Table.

[Svend_Dellepude]

What Yoda said. I do not run a dedi-server but I did sign up for the Locerf and was sorry to realize that i didn't come true. So anything to make an event like that come true is fine by me.

Does anyone knows if a largescale event like this has been done on LAN with succes?

And also thanks to all in planning this event even though it go quite the way it was planned.

 

Svend

[104th_Crunch]

How many events did NOT have technical problems in the start? (This is meant to be positive critics)

 

All of the 169th events. Why? because the missions used were the missions in regular rotation on the server, tested many times.

 

edit: I am sure Redflag will get there and soon. Many RFs were flown with no real technical problems as well. As Hajduk pointed out all Nucleus missions ran great and the 51st had CI running great. Just a matter of stress testing. Maybe also introduce a minimum 200 ping rule.

Edited June 22, 2010 by Crunch

[X-man]

How many events did NOT have technical problems in the start?

 

Quite a few actually ;)

[Breakshot]

Maybe also introduce a minimum 200 ping rule.

 

Sorry but that would be extremely unfair for many pilots located outside EU and US, including myself... At best i get around 220-250 to 51st server for instance, sometimes over 300 on bad days, but it never affected anything as long as its stable enough...

 

Ping is not the issue as long as it stays under 400, IMO... The issue here is having the server and clients overloaded with unnecessary things... We must run the next event with stock settings and IC to prevent additional stuff from breaking the server... and go from there if it works... its quite simple really... i dont think any pre-testing is gonna be needed either, as its hard to get enough pilots for those anyway...

Edited June 22, 2010 by Breakshot