fixed Invalid Ballistics Objects being created and not cleaned up resulting in FPS impact.

[rurounijones]

  

Summary:

DCS World is creating invalid ballistics objects which are not cleaned up. These ballistics objects persist until mission restart and have a significant impact on the performance of Multiplayer servers, measured in "Server FPS" or the number of simulation frames per second that the server is processing.

Background:

Ballistics objects are spawned whenever a rapid fire weapon starts firing. They may also be spawned when cluster munitions are dispensed but I have been unable to confirm that. Ballistics Objects tend to have an impact on Server FPS which depends on the power of the server, the number of objects and, apparently, the number of connected clients. This is because the server has to spend time calculating the trajectories and other properties (Collisions etc.) of these ballistic objects.

Here is an example of Ballistics Objects being spawned when rapid-fire weapons fire. You can see that the number of Ballistics objects in the mission spike when a shooting event starts and the resulting impact on server FPS. You can also see that the number drops back to 0 as the ballistics objects expire.

(Note: The below graphs come from a liberation mission running with only AI running on a home-server)


 

Bug:

There are times, however, when Ballistics objects increase without an associated shooting event (Or any other event that I can find).

You can also see that after this jump the objects are not cleaned up and the number of ballistics objects in-mission steadily accumulates.

 

If we look at these objects we can see that they are invalid and they are always identical aside from the main ID. Their type is all 0, their coordinates are all 0 and the lat/lon is always at map origin, the country is 99 which is not in the country enum. Clearly these are not something that should exist.

	"33584385": {
		"Pitch": 0,
		"Type": {
			"level3": 0,
			"level1": 0,
			"level4": 0,
			"level2": 0
		},
		"Country": 99,
		"Coalition": "Enemies",
		"Flags": {
			"Jamming": false,
			"IRJamming": false,
			"Born": false,
			"Static": false,
			"Invisible": false,
			"Human": false,
			"AI_ON": true,
			"RadarActive": false
		},
		"Name": "",
		"Position": {
			"y": 0,
			"x": 0,
			"z": 0
		},
		"Heading": 6.2831854820251,
		"LatLongAlt": {
			"Long": 34.265515188456,
			"Lat": 45.129497060329,
			"Alt": 0
		},
		"CoalitionID": 0,
		"Bank": 0
	}

I have also noticed that these invalid objects tend to (but not always) spawn 76 objects at a time as evidenced below (Note: This graph is from a Hoggit Georgia At War session).



The trackfile for this Liberation mission can be found in the attached `liberation-mission.trk` which is tiny because, I assume, there is only AI.

Impact:
The accumulation of these Ballistics Objects has a deleterious impact on the Hoggit multiplayer servers (And potentially others, but Hoggit is what I was analysing) due to the fact that the server still has to assign time to process them even though they are doing nothing. Here is a graph of a typical mission on Hoggit Georgia At War. You can see that the server FPS steadily decreases as the number of Ballistics objects increases. I could find no other metrics that shows such a correlation.

You can also see that the number of players also has an impact on the server FPS in that the FPS recovers very slightly as player count drops at the end. I am not sure if the number of players amplifies the impact of the ballistics objects but my assumption is yes because on a server with very few players the impact is not as great.

(Note: The below graphs come from the Hoggit "Georgia At War" server)

As well as a corresponding increase in CPU usage:



You can see here that the number of invalid Ballistics objects is in the thousands. However I have also observed that this depends on which of the two missions that make up "Georgia At War" is running (P1 or P2). In one mission the Invalid ballistics objects consistently numbers in the hundreds and one in the thousands. However even just a few hundred ballistics objects has a massive impact as well.

I have observed this behaviour across about 20 different mission sessions.

Track files for some GAW sessions with this ballistic object accumulation can be found here: https://drive.google.com/drive/folders/1-_Ae-h6dl5s2k9v1FWwfkME0OBOflbWt (Size warning. 300-400MB)

Request:
Eagle Dynamics to find the root cause for these invalid ballistics objects being spawned and fix it.
 

 

liberation-mission.trk

Edited September 3, 2022 by rurounijones
Added more information and clarifications

[sandboxcode]

Looks like a pretty serious issue I hope they can get eyes on this!

[StandingCow]

Nice, hopefully ED can investigate and find the cause.  Anything to help MP performance is a win for everyone.

[Harlikwin]

Yeah this would be a major boon if it got fixed for MP servers. 

[BIGNEWY]

7 hours ago, rurounijones said:

  

Summary:

DCS World is creating invalid ballistics objects which are not cleaned up. These ballistics objects persist until mission restart and have a significant impact on the performance of Multiplayer servers, measured in "Server FPS" or the number of simulation frames per second that the server is processing.

Background:

Ballistics objects are spawned whenever a rapid fire weapon starts firing. They may also be spawned when cluster munitions are dispensed but I have been unable to confirm that. Ballistics Objects tend to have an impact on Server FPS which depends on the power of the server, the number of objects and, apparently, the number of connected clients. This is because the server has to spend time calculating the trajectories and other properties (Collisions etc.) of these ballistic objects.

Here is an example of Ballistics Objects being spawned when rapid-fire weapons fire. You can see that the number of Ballistics objects in the mission spike when a shooting event starts and the resulting impact on server FPS. You can also see that the number drops back to 0 as the ballistics objects expire.

(Note: The below graphs come from a liberation mission running with only AI running on a home-server)


 

Bug:

There are times, however, when Ballistics objects increase without an associated shooting event (Or any other event that I can find).

You can also see that after this jump the objects are not cleaned up and the number of ballistics objects in-mission steadily accumulates.

 

If we look at these objects we can see that they are invalid and they are always identical aside from the main ID. Their type is all 0, their coordinates are all 0 and the lat/lon is always at map origin, the country is 99 which is not in the country enum. Clearly these are not something that should exist.

	"33584385": {
		"Pitch": 0,
		"Type": {
			"level3": 0,
			"level1": 0,
			"level4": 0,
			"level2": 0
		},
		"Country": 99,
		"Coalition": "Enemies",
		"Flags": {
			"Jamming": false,
			"IRJamming": false,
			"Born": false,
			"Static": false,
			"Invisible": false,
			"Human": false,
			"AI_ON": true,
			"RadarActive": false
		},
		"Name": "",
		"Position": {
			"y": 0,
			"x": 0,
			"z": 0
		},
		"Heading": 6.2831854820251,
		"LatLongAlt": {
			"Long": 34.265515188456,
			"Lat": 45.129497060329,
			"Alt": 0
		},
		"CoalitionID": 0,
		"Bank": 0
	}

I have also noticed that these invalid objects tend to (but not always) spawn 76 objects at a time as evidenced below (Note: This graph is from a Hoggit Georgia At War session).



The trackfile for this Liberation mission can be found in the attached `liberation-mission.trk` which is tiny because, I assume, there is only AI.

Impact:
The accumulation of these Ballistics Objects has a deleterious impact on the Hoggit multiplayer servers (And potentially others, but Hoggit is what I was analysing) due to the fact that the server still has to assign time to process them even though they are doing nothing. Here is a graph of a typical mission on Hoggit Georgia At War. You can see that the server FPS steadily decreases as the number of Ballistics objects increases. I could find no other metrics that shows such a correlation.

You can also see that the number of players also has an impact on the server FPS in that the FPS recovers very slightly as player count drops at the end. I am not sure if the number of players amplifies the impact of the ballistics objects but my assumption is yes because on a server with very few players the impact is not as great.

(Note: The below graphs come from the Hoggit "Georgia At War" server)

As well as a corresponding increase in CPU usage:



You can see here that the number of invalid Ballistics objects is in the thousands. However I have also observed that this depends on which of the two missions that make up "Georgia At War" is running (P1 or P2). In one mission the Invalid ballistics objects consistently numbers in the hundreds and one in the thousands. However even just a few hundred ballistics objects has a massive impact as well.

I have observed this behaviour across about 20 different mission sessions.

Track files for some GAW sessions with this ballistic object accumulation can be found here: https://drive.google.com/drive/folders/1-_Ae-h6dl5s2k9v1FWwfkME0OBOflbWt (Size warning. 300-400MB)

Request:
Eagle Dynamics to find the root cause for these invalid ballistics objects being spawned and fix it.
 

 

liberation-mission.trk 857.11 kB · 2 downloads

 

I will ask the team about this. 

thanks

[Sealpup]

If these objects all share a location, type, and country, would it not be possible to create a 'garbage collector' script to periodically remove them? A bandaid fix until the root cause can be addressed.

[HeavyGun1450]

18 hours ago, BIGNEWY said:

I will ask the team about this. 

thanks

Please make a rapid hotfix for this. It is destroying every single multiplayer servers FPS greatly currently.

[BIGNEWY]

7 hours ago, HeavyGun1450 said:

Please make a rapid hotfix for this. It is destroying every single multiplayer servers FPS greatly currently.

We need to investigate it first and find the reason, it is not clear at the moment, it could be something in the missions for example. If we find something we will let you all know. 

thanks

 

[rurounijones]

13 hours ago, Sealpup said:

If these objects all share a location, type, and country, would it not be possible to create a 'garbage collector' script to periodically remove them? A bandaid fix until the root cause can be addressed.

The community does not have access to any APIs that can garbage collect these objects as far as I am aware.
 

2 hours ago, BIGNEWY said:

We need to investigate it first and find the reason, it is not clear at the moment, it could be something in the missions for example. If we find something we will let you all know. 

thanks

I fully agree that it will probably take time since debugging can be a pain and hope ED do not rush the investigation and potentially come out with a non-fix or break something in a rush to resolve the issue.

However I hope that "It might be something in the mission" will not be a position that ED take as a reasoning for doing nothing since the DCS World Platform should not allow invalid objects to be created via its APIs in the first place. It should instead return an error if it is incorrectly called rather than silently create invalid objects.

* If the problem lies within DCS world and can be fixed then great!
* If the problem lies in the scripting involved in the missions then the DCS World API that is allowing these invalid objects be created should have validation added and return an error instead. Not just in this case, but in general.

Edited September 5, 2022 by rurounijones

[BIGNEWY]

4 minutes ago, rurounijones said:

I fully agree that it will probably take time since debugging can be a pain and hope ED do not rush the investigation and potentially come out with a non-fix or break something in a rush to resolve the issue.

However I hope that "It might be something in the mission" will not be a position that ED take as a reasoning for doing nothing since the DCS World Platform should not allow invalid objects to be created via its APIs in the first place. It should instead return an error if it is incorrectly called rather than silently create invalid objects.

* If the problem lies within DCS world and can be fixed then great!
* If the problem lies in the scripting involved in the missions then the DCS World API that is allowing these invalid objects be created should have validation added and return an error instead. Not just in this case, but in general.

correct, no one knows yet, it needs to be investigated. 

[DragonSoulkin]

In the meantime what I'm wondering is what is our suggested course of action as players trying to continue having a multiplayer experience?

[Rewis.C]

What program are you using to see those statistics?

[Rinz1er]

On 9/7/2022 at 8:04 PM, Rewis.C said:

What program are you using to see those statistics?

Looks like grafana UI and https://github.com/DCS-gRPC/Telemachus

Edited September 9, 2022 by Rinz1er

[Tom P]

Is this why the multiplayer servers have been going from working servers to all the sudden things being broke? Our servers have all the sudden broke and our mission editors don't want to keep making changes just for it to be broke again in a future patch. 

[FupDuck]

On 9/5/2022 at 4:39 PM, DragonSoulkin said:

In the meantime what I'm wondering is what is our suggested course of action as players trying to continue having a multiplayer experience?

Patience and a positive attitude?

 

These things take time to fix properly.

[rurounijones]

@BIGNEWY
Do you know if anything was included in DCS 2.7.18.30348 Open Beta related to this report?

 

[BIGNEWY]

1 minute ago, rurounijones said:

@BIGNEWY
Do you know if anything was included in DCS 2.7.18.30348 Open Beta related to this report?

 

Not to my knowledge, the team is having problems reproducing this at the moment, and tests are not conclusive at the moment. 

They continue to investigate. 

Please let me know if you see any change your end. 

thanks

[Tom P]

22 hours ago, BIGNEWY said:

Not to my knowledge, the team is having problems reproducing this at the moment, and tests are not conclusive at the moment. 

They continue to investigate. 

Please let me know if you see any change your end. 

thanks

still broke. After server restarts ground units either spawn dead or not at all. You need to go onto the front end and manually restart the server to get it to work. 

 

[BIGNEWY]

8 minutes ago, 162nd Pete said:

still broke. After server restarts ground units either spawn dead or not at all. You need to go onto the front end and manually restart the server to get it to work. 

 

This is a different issue from the one above, which is ballistics.

[Mike_Romeo]

Intrestring watch that might have something to do with the bug

 

[rurounijones]

On 9/21/2022 at 4:37 PM, BIGNEWY said:

Please let me know if you see any change your end.

Confirmed still occuring on latest patch

[BIGNEWY]

thanks I will pass it on to the team

[Tom P]

soooo

[rurounijones]

@BIGNEWY
Do you know if anything was included in DCS 2.8.0.32066 Open Beta related to this report?

[rurounijones]

Ballistics objects are still accumulating in DCS 2.8.0.32066 Open Beta

Edited October 29, 2022 by rurounijones
remove rogue "null"