[REPORTED]Lag and desync after seeing a specific error spamming our logs

[jers]

Hello everyone.

We added the Supercarrier to our Persian Gulf mission today and things seem to go fine for a while but after about an hour or so we begin to see this error spamming out logs thousands of times over:

 

 

[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 WARNING EDOBJECTS: RegMapStorage has no more IDs[/font][/color]
[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42[/font][/color]
[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 WARNING EDOBJECTS: RegMapStorage has no more IDs 
[/font][/color]
[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42[/font][/color]

We've tried reducing the number of units in the mission taking a hint from that error, but so far it hasn't seemed to help. Is there any way an engineer can tell us exactly what triggers this error so that we might be able to fix whatever it is that's triggering it?

 

 

 

Thanks!

 

Zelly

[jers]

Update to this, with some research I found that it looks like there's an upper limit of 4000 groups, inspecting our logs this problem happens every time at some point int he mission when the supercarrier is respawning crewmen.

 

2020-05-2221:28:00.356WARNING FLIGHT: procCrewAssignAircraft client id 4294967295, crew_id 2, hot_start 0

2020-05-2221:28:00.356WARNING FLIGHT: crew 2 wCarrierCrew::reset_LA()

2020-05-2221:28:00.356WARNING EDOBJECTS: RegMapStorage has no more IDs

2020-05-2221:28:00.356ERROR EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

2020-05-2221:28:00.356WARNING FLIGHT: carrier technician [2: shooter] created, id 4

2020-05-2221:28:00.356WARNING EDOBJECTS: RegMapStorage has no more IDs

2020-05-2221:28:00.356ERROR EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

2020-05-2221:28:00.356WARNING EDOBJECTS: RegMapStorage has no more IDs

2020-05-2221:28:00.356ERROR EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

2020-05-2221:28:00.356WARNING EDOBJECTS: RegMapStorage has no more IDs

2020-05-2221:28:00.356ERROR EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

2020-05-2221:28:00.356WARNING EDOBJECTS: RegMapStorage has no more IDs

2020-05-2221:28:00.356ERROR EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

[bullet0]

Exact same error on the 104th server after 4-5 hours it increases frequency causing problems

[Pikey]

That's the DCS engine runnign out of ID's for new spawns. You've done something horrible to produce over 4000 groups in that mission. Fix that and you will be fine.

[EDIT: ah I see you say this is the SC producing spawns? very bizzarre. Only way to test is to start a mission, do 4000 spawns, count the index it got too, then repeat after 30 minutes of some basic AI mimicking whatever it is that you are doing in normal play and doing the 4000 spawn again to see the new (and supposedly less) index it reaches, minus what you spawned for the test, to conclude the limit is nothing to do with AI

Edited May 24, 2020 by Pikey

[Pikey]

I tried to force this by spawning 4000 groups and then having some spawns on the carrier which returned and a menu spawn to add more, counting up. In the logs and I could not replicate the 4000 hard limit. The only way I could get that error message was doing a good 5000 at once and even then it went away in the logs pretty quickly and spawns could continue. I tried batches of a few thousand after each other and various combinations but gave up after the 4200 mark.

 

 

 

So I attached the mission and you can have a look in your logs and play about with it, but basically I can't reproduce, so something else is going on.

 

 

 

I'd be interested to see the logs and find out what you are running on these missions to cause this. If you have good steps to reproduce I'd also look for you.

spawnCountTest.miz

[BIGNEWY]

Thanks I have asked the team to take a look.

[BIGNEWY]

Hello everyone.

We added the Supercarrier to our Persian Gulf mission today and things seem to go fine for a while but after about an hour or so we begin to see this error spamming out logs thousands of times over:

 

 

[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 WARNING EDOBJECTS: RegMapStorage has no more IDs[/font][/color]
[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42[/font][/color]
[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 WARNING EDOBJECTS: RegMapStorage has no more IDs 
[/font][/color]
[color=#B9BBBE][font=Consolas]2020-05-23 02:30:57.535 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42[/font][/color]

We've tried reducing the number of units in the mission taking a hint from that error, but so far it hasn't seemed to help. Is there any way an engineer can tell us exactly what triggers this error so that we might be able to fix whatever it is that's triggering it?

 

 

 

Thanks!

 

Zelly

 

Can we see a full server log showing this.

 

How many units are you spawning?

 

Also can you attach the mission.

 

thanks

[BIGNEWY]

Exact same error on the 104th server after 4-5 hours it increases frequency causing problems

 

Hi bullet0

 

please attach a server log showing this, we will take a look.

 

Also can you tell us the server spec.

 

thanks

[BIGNEWY]

Hi all

 

I have reported RegMapStorage has no more IDs in correlation to server FPS, once the team take a look I will feed back to you all.

 

Thanks

[Pikey]

I came back to this remembering something. I don't think a 4000 or so limit on group ID's is a major problem, although it has been, where I saw it before it was fixed, because it was overzelous scripting that was occurring without the scripters knowledge.

 

 

But what IS potentially concerning is if the deck crew on SC are using them up way faster than we thought. I haven't been able to tell if they are, and I think the assumption/fear is mostly based on the arrival of SC and the log entries folks see, where the ID appears to be used up.

 

2020-05-30 11:15:57.873 WARNING FLIGHT: carrier technician [4: shooter] created, id 8 2020-05-30 11:15:57.873 WARNING FLIGHT: carrier technician [4: director] created, id 9 2020-05-30 11:15:57.873 WARNING FLIGHT: carrier technician [4: white1] created, id 10

Are these crew meant to be accessible later? I believed they were just eye candy.

Edited May 30, 2020 by Pikey

[robgraham]

You wanted a log Newy, here you go.. this one doesn't spawn ground units after the start the only thing that respawns are Air units for CAP etc.

 

13 hours into the mission (that we'd hope could run for long periods due to reduction in coding etc etc ) we started to see this

 

2020-05-31 01:37:53.230 WARNING LOG: 23 duplicate message(s) skipped.
2020-05-31 01:37:53.230 WARNING EDOBJECTS: RegMapStorage has no more IDs
2020-05-31 01:37:53.230 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42
2020-05-31 01:37:53.289 WARNING EDOBJECTS: RegMapStorage has no more IDs

 

every second just spamming the logs and a FPS decrease to go along with it, why is there a limit like this in this day and age and if it's 4000 like people are claiming does that include Ordiance spawns? Because if it does then that numbers massively low for what people do these days.. and if the SC etc is spawning and despawning stuff, I've attached the log. you'll see it start at the time stamp given above. Mission is attached, should run even with sanitised files etc as no server side loading etc etc etc taking place on this one.

dcs.rar

Persia_Aflamev0_65.miz

Edited May 31, 2020 by robgraham

[BIGNEWY]

Thanks for the log, the team is looking into it.

[NeedzWD40]

I'd like to add some additional information on this issue:

 

 

I'm attaching a WWII Normandy scenario that is scripted to last for 6 hours with AI aircraft groups respawning only, no ground vehicle respawns, and AI aircraft set to respawn every 5 minutes. Some groups have waypoints that change depending on objective status, but other than that they fly in a linear fashion. There's a master script that runs every minute to check the time, objectives, and aircraft group status, then adds groups to a table if it's time to respawn them. A flag is then tripped and a separate script attached to a continuous trigger runs the actual spawning script. Through this mechanic, I keep track of how many groups are spawned via the script.

 

 

Now, the interesting part is after approximately one hour of activity, I start to get out of group errors, even though my spawn counter has only gone up to 13 groups. Assuming the IDs error applies to units, that's only 26 units, as there's two aircraft per group (except for two groups that only spawn every hour, where there's four aircraft in these groups). Groups still spawn, but the out of groups error still spams the log.

 

 

Using env.info, I track various script status messages:

 

 

2020-06-05 21:44:10.734 INFO    Lua: Lua CPU usage: metric: average mission execution: 4.4816 %
2020-06-05 21:46:10.603 INFO    SCRIPTING: bFZAb Air spawner OFF
2020-06-05 21:46:10.603 INFO    SCRIPTING: bFZAb Masterloop Heartbeat Time: 66 Total spawned groups: 12
2020-06-05 21:47:03.033 INFO    Lua: Lua CPU usage: metric: average mission execution: 4.3806 %
2020-06-05 21:47:38.584 ERROR_ONCE DX11BACKEND: texture 'p51d_spec_fuz_rear' not found. Asked from 'NGMODEL'
2020-06-05 21:47:42.338 INFO    SCRIPTING: bFZAb Blue aircraft spawner
2020-06-05 21:47:42.338 INFO    SCRIPTING: bFZAb Inserting B_P51DCAP01 to build list
2020-06-05 21:47:42.338 INFO    SCRIPTING: bFZAb Air spawner ON
2020-06-05 21:47:42.338 INFO    SCRIPTING: bFZAb Masterloop Heartbeat Time: 67 Total spawned groups: 12
2020-06-05 21:47:43.349 INFO    SCRIPTING: bFZAb Running Air Spawner
2020-06-05 21:47:43.559 INFO    SCRIPTING: bFZAb Spawned air group B_P51DCAP01
2020-06-05 21:48:11.294 INFO    Lua: Lua CPU usage: metric: average mission execution: 5.2583 %
2020-06-05 21:49:06.735 INFO    SCRIPTING: bFZAb Air spawner OFF
2020-06-05 21:49:06.735 INFO    SCRIPTING: bFZAb Masterloop Heartbeat Time: 68 Total spawned groups: 13
2020-06-05 21:49:59.306 ERROR   SOUND: can't load wave: "speech\eng\common\atc\callsign\carpiquet"
2020-06-05 21:50:06.732 INFO    SCRIPTING: bFZAb Air spawner OFF
2020-06-05 21:50:06.732 INFO    SCRIPTING: bFZAb Masterloop Heartbeat Time: 69 Total spawned groups: 13
2020-06-05 21:50:42.764 WARNING EDOBJECTS: RegMapStorage has no more IDs
2020-06-05 21:50:42.764 ERROR   EDOBJECTS: Failed assert `false` at Projects\edObjects\Source\Registry\RegMapStorage.cpp:42

 

 

"Masterloop Heartbeat" measures the current running minute of the script since it started, "Air spawner" indicates whether or not the group spawn flag has been set, "Inserting X into build group list" is indicating what group is being added to the spawner array. The spawner script then indicates it is running and what group(s) it spawned. Note that this script will run only once per minute, so if multiple groups are to be spawned, they will have to wait a minute for the next iteration. Finally, "Total spawned groups" is exactly what it says on the tin, though this value tracks how many groups have been added to the spawn array/list rather than spawned groups directly through the spawning script.

 

 

This scenario was ran for 108 minutes in singleplayer (testing new P-47 groups replacing P-51s), so this particular issue doesn't seem specifically constrained to multiplayer situations. It also seems to be more pertinent with WWII assets versus modern ones. I've attached my (quite large) log from this run for analysis.

pegasus_gamble_nm_ob256v0.miz

dcs_log_pegasus_gamble_060520.zip

[BIGNEWY]

Thanks, we are looking at solutions for this issue.

[riojax]

On 6/6/2020 at 10:42 AM, BIGNEWY said:

Thanks, we are looking at solutions for this issue.

Any update on this, we are suffering this problem in our servers too.