fixed Invalid Ballistics Objects being created and not cleaned up resulting in FPS impact.

[Flappie]

My mistake, I mistook this bug with another one. As an apology, I'll try hard to reproduce this one.

[Flappie]

I think I found something, thanks to @abelian's amazing tool.

In @rurounijones's AI-only track, at 18:35:25, the Kuznetsov fires 16 "P-500" missiles (or "P-700", it has two differents names in DCS). They travel a long way, aiming at blue ships. They eventually all get shot down by SM-2s. But... somehow they're still alive.

EDIT: I need a few more tests before reporting this, but I'll definitely report it.

Edited January 11, 2023 by Flappie

[Flappie]

I tried to reproduce this issue in a simple SP mission: one Kuz and one Arleigh Burke. I tried the new and old Kuz, but they both shoot unit objects, not ballsitic objects, which means I cannot reproduce the issue on my own for now, only when using the track in the OP.

Was something special done in the mission for it to happen?

Time to go to bed. More testing tomorrow.

Edited January 11, 2023 by Flappie

[rurounijones]

13 hours ago, Flappie said:

Was something special done in the mission for it to happen?

The AI only track is a self-contained liberation generated mission so the only scripting that may be involved will be in the mission file itself. I am not aware of anything liberation does that would cause wierdness, I think their scripting is imited to writing stuff out when the mission closes. I am not sure if could be scripting or anything outside of the ED generated code that could be causing the issue directly since there are no APIs available for creating or managing ballistics objects.

Obviously something is causing this, but I think the root cause must lay somewhere in the built-in ED code, not any 3rd party scripting.

I don't think it is the ship missiles are the root cause since the ballistics count can go very high on GAW servers where ships are not firing at all. The worst case is in the thousands. But I wonder if it could be a pointer in that some long-lived ballistics objects are not being deleted properly and an invalid object takes their place? However I see the ballistics count increase almost immediately sometimes on server start so again I am not sure it could be related to existing ballistics objects.
 

Edited January 11, 2023 by rurounijones

[Harlikwin]

14 hours ago, Flappie said:

I tried to reproduce this issue in a simple SP mission: one Kuz and one Arleigh Burke. I tried the new and old Kuz, but they both shoot unit objects, not ballsitic objects, which means I cannot reproduce the issue on my own for now, only when using the track in the OP.

Was something special done in the mission for it to happen?

Time to go to bed. More testing tomorrow.

 

Thanks for trying to get this figured out. IDK if the shadows thing on ECW might also be a clue, but what happens there is when you are looking at the front line after a while the CPU frametimes get absurdly bad in VR. Like 50ms+ while the GPU frametimes are lik 5-7ms. However at the start of the mission the CPU frametimes are like 10-15ms. The interesting bit is if you turn off shadows entirely those 50ms CPU frametimes return to the normal 10-15ms. GPU frametimes are steady at like 5-7ms regardless. 

I think the game is trying to shade all those non-existent/not despawned objects. 

[Flappie]

4 hours ago, rurounijones said:

I don't think it is the ship missiles are the root cause since the ballistics count can go very high on GAW servers where ships are not firing at all. The worst case is in the thousands. But I wonder if it could be a pointer in that some long-lived ballistics objects are not being deleted properly and an invalid object takes their place? However I see the ballistics count increase almost immediately sometimes on server start so again I am not sure it could be related to existing ballistics objects.

I never said the problem was limited to the Kuz. I said I found something. Now, I'm digging.

Edited January 11, 2023 by Flappie

[Flappie]

I was wrong, the 16 objects are P-500 missiles fired by the Moscow, not the Kuz (which fires P-700).

Here's the detailed count for ballistic objects in the AI track (38 first minutes).

18:33:40 -> 16 P-500 fires by Moscow (at this time, 0 ballistic objects detected)
18:35:16 -> 19 ballistic objects detected at once = A-10C_2 gets hit by SA-19
18:35:43 -> 16 ballistic objects detected left (3 died) = A-10C_2 crashes
18:37:46 -> 17 (+1) M270 MLRS shooting
18:37:50 -> 18( +1) M270 MLRS shooting
18:37:53 -> 19 (+1) M270 MLRS shooting
18:37:56 -> 20 (+1) M270 MLRS shooting
18:38:59 -> 21 (+1) M270 MLRS shooting
18:38:03 -> 22 (+1) M270 MLRS shooting
18:38:06 -> 23 (+1) M270 MLRS shooting
18:38:10 -> 22 (-1) M270 MLRS shooting
18:38:12 -> 23 (+1) M270 MLRS shooting
...
18:38:35 -> back to 16 (and they stay forever)

Theory 1: the A-10 getting shot creates 3 ballsitic objecs, but then the 16 P-500 missiles are instantly turned into ballsitic objects : 16+3 = 19, then 16 again once A-10C_2 crashes.

Theory 2: the A-10 getting shot created 19 ballistic obects, among them 16 last forever and 3 dies.

In any case, the apparition of these 16 eternal ballsitic objects is triggered at the very moment the A-10C_2 gets split into pieces by an SA-19 missile.

Edited January 11, 2023 by Flappie

[Flappie]

@rurounijones You were right: the ship's missiles had nothing to do with it. I was able to reproduce this issue is a very short track with only a handful of units: some A-10C_2 and some SA-19. It seems the mysterious ballistic objects are nothing but A-10C_2 pieces taken out by missiles.

I tested some other aircraft: "old" A-10C, F-16C and Apache don't spread such objects.

Issue reported.

Edited January 18, 2023 by Flappie

[Flappie]

UPDATE: the A-10C_2 debris had nothing to do with this. @Hobel found out that it was caused by the jettisoned racks (AI units jettison their racks as soon as they take important damage).

It seems all jettisoned racks are affected. So far we identified these:

LAU-10
LAU-131
LAU-68
BRU42
LAU88
M299
LAU117

We are yet to find what simultaneously creates 76 ballistic objects on the GaW server (76 / 2 = 38 , 38 / 2 = 19 - this does not ring a bell). We need a MP track showing the 76 objects (SP is OK too, since the issue affects both SP and MP).

[Moezilla]

I know right now you need some tracks but I just wanted to add my separate confirmation regarding the jettisoned objects. This time it is from the MiG-21.
Data was gathered on Enigma's Dynamic Cold War server using @abelian's tool.
Two large groups of objects (58 and 66) seem to have been created by two separate MiG-21s who also seem to have been in close proximity. The groups of objects were created within roughly 2 seconds of each other and proceed to lose altitude until they reach 0 units of altitude and then their position shifts to the map origin point where they remain, as was reported by @rurounijones. The objects had been in existence for over 23 minutes when I left the server.
 

[BIGNEWY]

We will take a look at the mig-21 thanks

[Moezilla]

@BIGNEWY@Flappie
I recreated the issue with the MiG-21 with a simple mission. When the MiG-21 jettisons the UB-32 pods with the S-5 rockets, 66 objects are created, 2 (the pods) are cleaned up but 64 objects are not (likely the rockets) and are shifted to the map origin point as in the original report when they reach the "ground".

Thinking about the 76 objects in the GAW mission it is likely that the pods and other stores which are jettisoned together are counted in that total but only the rockets (hydra in that case) are not cleaned up correctly. So something like 8xpods + 56 rockets + some other stores could get you to 76 initial objects created.

I hope these latest discoveries will lead to a comprehensive fix and the players won't have to create a track and report for every pod/munition combination.

type99_jettison_test.miz

Edited January 18, 2023 by Moezilla

[BIGNEWY]

thanks again we will add to the list to check

[Hobel]

vor 5 Stunden schrieb Moezilla:

Thinking about the 76 objects in the GAW mission it is likely that the pods and other stores which are jettisoned together are counted in that total but only the rockets (hydra in that case) are not cleaned up correctly. So something like 8xpods + 56 rockets + some other stores could get you to 76 initial objects created.

Good hint

 

 

EDIT:

Okay my mistake I looked at the mission again carefully, an apache dropped racks.

Edited January 18, 2023 by Hobel

[Flappie]

@Moezilla Thanks for the .miz file. I can confirm what you said. As I said above, it seems ALL jettisoned racks are affected. A fix is already in the works for jettisoned racks.

Now, maybe some other things are also causing these eternal ballistic objects. I tried countermeasures, but they are not (ATGM ammo is another source of eternal ballistic objects but it's reported already).

[rurounijones]

> Now, maybe some other things are also causing these eternal ballistic objects. I tried countermeasures, but they are not (ATGM ammo is another source of eternal ballistic objects but it's reported already).

I think there probably is something else due to the number discreprencies between GAW P1 and GAW P2 but it might be better to wait until the current known issues are fixed then do some more recording of trackfiles between the two (i.e. if P1 stops accumulating completely and P2 is still accumulating) which may make finding other causes easier.

[Mike_Romeo]

Apperently it was fixed with the latest open beta

[Moezilla]

That particular issue seems to have been fixed, and that's great,  but more testing will have to be done on live servers to be sure that all sources of the objects have been found.

[Flappie]

@rurounijones I'm looking forward to reading your feedback.

[rurounijones]

The Good

After a few mission cycles I beiieve this issue has been fixed. Ballistics are not accumulating, server FPS is not degrading and CPU usage is not climbing. It is possible that this was the sole root cause of the ballistics leak and there are no others. If I come across any then I will create a new thread.

So congrats to @Flappie@abelian@Moezilla, the others who helped, and the ED staffers who fixed and tested.

The Bad
While I am happy the issue has been resolved I still believe that ED's handling of this issue has overall fallen short of acceptable standards and I want to go into detail here so that, maybe, things can improve.

Server Owner <-> ED Relationship
I posted this issue on 2022/09/01 however the general issue of a ballistics object leak was known about by the Hoggit admins for a long time before that with posts in the Discord describing the general issue from 2022/06/01. Now I don't know if anything was reported or not but, having spoken to many different server admins, the general concensus I hear is "It is a waste of time to report server issues to ED" along with a general lack of support, debug tooling etc.

This is an awful situation. If a server is having an issue that usually means all the clients on that server are having a suboptimal experience. In the case of this issue it meant players suffering lag, warping and otherwise unfavourable conditions. How many ED customers on the various multiplayer severs have left with a sour experience due to this bug in the course of the 6 months or so it took to resolve it? I am willing to bet that it is thousands of customers and thousands of hours of flying time where customers have been left with a bad taste in their mouth at the end of a session .

My suggestion to ED to improve this situation is:

1. Provide a formal way for mutliplayer server admins of servers with large player bases to contact you to report issues. Either a forum or a discord channel explicitly for communicating issues to ED and collaborating on fixing them. You have stats on which servers are heavily populated I am sure so invite their admons. Admins having to rely on "Try pinging @NineLineor @BIGNEWYand hope for the best" doesn't cut it.

2. Having created this formal communication method. Listen to them and accept that things like "please provide trackfile" is not always feasible on servers with scripts that run for hours with 10s of players on them where an issue may be inconsistent. Work with them collaboratively to try and find the issues which leads me to the last point:

3. Provide them with the tools they need to be able to perform this kind of investigation. I know this has been requested because multiple people have complained to me about these requests being ignored.

The handling of this issue by ED, aka "Cannot Reproduce" & Radio Silence
I am going to go out on a limb here and say that this bug report was about as good as ED can expect without deep-diving and finding the root cause itself. It contained lots of information, it pointed to exact syptoms. It included 4 different trackfiles from two completely different setups. Including an AI only tiny trackfile.

Despite this the response was "Cannot reproduce". But when @Flappiestarted looking into it he appears to have been able to reproduce it from the small AI trackfile I provided in short order.

So the questions that ED needs to answer, at least to itself is "Why couldn't ED staff reproduce something that was reproduced by a volunteer in short order using submitted data.". If ED cannot reproduce things via trackfiles then why should the community spend time and effort to provide them?

My next issue is that, once the "cannot reproduce" status was entered that was basically it. The issue languished for months and there were no updates; I had to literally contact ED staff on discord to try and get an update on what what was going on. Had @Flappienot taken the effort to root cause this I have no doubt that this would still be unresolved and ED customers would still be suffering. This ties into the linked post which I think ED needs to consider a lot more carefully.

Edited January 26, 2023 by rurounijones
speelung

[NineLine]

This issue is fixed, and while I understand the issue some people have with how issues are handled the truth is we are doing our best, and its not just me and BN doing it, but Testers like Flappie helping out. This thread is not for the discussion of that so I have hidden the off topic chatter and will close this one now as it is fixed.