DCS needs a build-in standardized, easy-to-use VR benchmark. ***EDIT: Proof of Concept added!

[Flextremes]

While for pancake (2D) simmers in DCS performance tuning is also a critical topic, it is imperative to do so with VR.

The stack to render DCS to your eye's retina is in VR so much more complicated than in 2D: we VR users are constantly tweaking and tuning. Frame-times spikes, jitter, fps swings, CPU bottlenecks, and GPU limits have a massive impact on our VR experience.

Currently, performance analysis is so cumbersome, we are really missing a standardized, consistent, precise, and easy-to-use solution to measure and compare performance in VR.

I believe we really need a simple, in-game, standardized benchmark solution to analyze (community build) replay tracks with fixed views for consistency. This allows us to easily compare fps statistics and frame-time data to tweak, tune and troubleshoot DCS the VR stack. Importantly, it would be essential to integrate performance statistics from the render stacks like OpenXR and SteamVR.

****EDIT****

I created a proof of concept below, you can use any track you like:

 

Edited March 15, 2023 by Flextremes

[The_Nephilim]

the only problem with that is it would need to be a free model and those models hardly take much resources. I have seen a benchmark made like you stated at the DCS Website in user files I think it used a su-25 but that was quite a few patches ago. no telling if it would still be good.

 

Can you play tracks of modules you do not own?

[pablofe]

4 hours ago, Flextremes said:

While for pancake (2D) simmers in DCS performance tuning is also a critical topic, it is imperative to do so with VR.

The stack to render DCS to your eye's retina is in VR so much more complicated than in 2D: we VR users are constantly tweaking and tuning. Frame-times spikes, jitter, fps swings, CPU bottlenecks, and GPU limits have a massive impact on our VR experience.

Currently, performance analysis is so cumbersome, we are really missing a standardized, consistent solution to measure and compare performance in VR.

I believe we really need a simple, in-game, standardized benchmark solution to analyze (community build) replay tracks with fixed views for consistency. This allows us to easily compare fps statistics and frame-time data to tweak, tune and troubleshoot DCS the VR stack. Importantly, it would be essential to integrate performance statistics from the render stacks like OpenXR and SteamVR.

 

 

I have been thinking of a project along the same lines

Creating AI missions with multiple assets and complexity, capturing the DCS settings as well as computer configuration, and reporting the results automatically to an online repository.

I have worked on similar industry graphics benchmarks many years ago so this project is on my list for this year.

-Pablo

[Glide]

One technique we used at MSFS is to spawn in with a specific plane at a specific airfield and measure response time.  You could choose a canned mission for the Su-25 or TF-51D on the Caucasus map, for example.

[GunSlingerOZ]

I have called for this for years. The suggestion is shot down by 99% of the community, who say ED shouldn't waste development resources on it. 

The thing is, performance tweaking is a HUGE part of setting up DCS. Most of us do it, VR users even more regularly. Getting everyone testing with identical methodology via an integrated benchmark would save so much pain. 

ED has never commented on building a benchmarking tool, but perhaps with all these new performance improvements, they'll spend a few days putting something simple together?

Edited March 15, 2023 by GunSlingerOZ

[Flextremes]

20 hours ago, The_Nephilim said:

the only problem with that is it would need to be a free model and those models hardly take much resources. I have seen a benchmark made like you stated at the DCS Website in user files I think it used a su-25 but that was quite a few patches ago. no telling if it would still be good.

 

Can you play tracks of modules you do not own?

I agree. I'm not sure you can.

However, modules can be installed and deactivated when you do not have a license. All modules already support the temporary license regardless. ED should just allow disabled modules to be used during replay and disable all inputs and view changes: this is essential for consistent benchmarking anyhow.

While it is dangerous to say something is easy to implement, this does not sound dramatically complicated. All ingredients are here: allow disabled modules in replay, disable all inputs and view changes/headset movements, and automatically start and stop frame time logging in OpenXR/SteamVR. It is that simple.

We can use external/3rd party tooling for analysis.

Edited March 15, 2023 by Flextremes

[Flextremes]

7 hours ago, pablofe said:

I have been thinking of a project along the same lines

Creating AI missions with multiple assets and complexity, capturing the DCS settings as well as computer configuration, and reporting the results automatically to an online repository.

I have worked on similar industry graphics benchmarks many years ago so this project is on my list for this year.

-Pablo

That would be awesome. (BTW I designed the benchmarking software that AMD used in their Epyc CPU keynote last year)

We could start simple, all ingredients are here:

• allow disabled modules in Replay;
• disable all inputs, automatically center view before start, and disable view changes/headset movements during track replay;
• and automatically start and stop frame time logging in OpenXR/SteamVR.
   

We can use external (community or freeware) tooling for analysis.  It is that straightforward. 

Edited March 15, 2023 by Flextremes

[WipeUout]

I think it is more complicated than this.  What airframe do you use?  What map?  SP or MP? Preferred scenario, bombing, dogfight?  And also, what are your HMD's options/features? 

Making a track and re-using it for your own benchmark, in accordance with the type of flying/map/airplane you prefer, is not that complicated.  You replay the track and then tweek your setting and replay.  With an integrated benchmark, you would change your setting, run the benchmark, then change settings and replay benchmark. No?

I think that a complete guide on all common settings (Steam, OXRTK, DCS) that is well maintained and include particulars per HMD would be far more useful.  The info is out there but all over the place.  Such a guide would be a time saver.   

[Flextremes]

1 hour ago, WipeUout said:

I think it is more complicated than this.  What airframe do you use?  What map?  SP or MP? Preferred scenario, bombing, dogfight?  And also, what are your HMD's options/features? 

Making a track and re-using it for your own benchmark, in accordance with the type of flying/map/airplane you prefer, is not that complicated.  You replay the track and then tweek your setting and replay.  With an integrated benchmark, you would change your setting, run the benchmark, then change settings and replay benchmark. No?

I think that a complete guide on all common settings (Steam, OXRTK, DCS) that is well maintained and include particulars per HMD would be far more useful.  The info is out there but all over the place.  Such a guide would be a time saver.   

It does not have to be.

If DCS allows any module for replay purposes only, the community can design tracks that are focussed on CPU or GPU or both. 

Guidance can be improved for sure: however there are already plenty of guides, but the world is continuously evolving and users have very different setups and even more different preferences.

No guide will ever take away the need for performance testing. How do you know something right or wrong, and what the impact is of a configuration change? 

Edited March 15, 2023 by Flextremes

[WipeUout]

5 minutes ago, Flextremes said:

the community can design tracks

A DCS track is an automatic recording of keystrokes and controllers input captured while flying a mission.  In order to fly a mission, you need the map and the aircraft module.  

If I understand what you propose, you would have community members creating the tracks with all possible combinations of airframe/map/mission type, and then have those tracks selectable as part of a benchmark?  Not simple nor easy.  Everyone has the liberty to create tracks that are relevant to their gameplay, more easy and flexible!

 

[Flextremes]

16 minutes ago, WipeUout said:

A DCS track is an automatic recording of keystrokes and controllers input captured while flying a mission.  In order to fly a mission, you need the map and the aircraft module.  

If I understand what you propose, you would have community members creating the tracks with all possible combinations of airframe/map/mission type, and then have those tracks selectable as part of a benchmark?  Not simple nor easy.  Everyone has the liberty to create tracks that are relevant to their gameplay, more easy and flexible!

 

I'm sorry, but it does not make any sense what your saying. You are over-complicating something that is not.

Example: fly one of the more complex missions on heavy duty maps (marianas) with the AH64. Fly and record the action for 2 minutes. You have a track. Share it with others. Done.

Additionally, I am sure DCS members love to design missions for performance testing and share them. This already has been done many times before.

 

Edited March 15, 2023 by Flextremes

[SharpeXB]

Realize that the track feature in DCS does not work and hasn’t for many years. Therefore it’s impossible to do this accurately even for yourself let alone have a standard track in the game or to share. That’s all you need to test for yourself anyways. Just make a track and rerun it as you adjust your settings. The trouble is if you try to do anything lengthy or complex, chances are the track will not end up showing the same thing every time since the feature is broken. 
Theres no point is asking for this from ED unless they fix the replay feature. Which they should. But apparently this is a big deal. 

[Flextremes]

5 hours ago, SharpeXB said:

Realize that the track feature in DCS does not work and hasn’t for many years. Therefore it’s impossible to do this accurately even for yourself let alone have a standard track in the game or to share. That’s all you need to test for yourself anyways. Just make a track and rerun it as you adjust your settings. The trouble is if you try to do anything lengthy or complex, chances are the track will not end up showing the same thing every time since the feature is broken. 
Theres no point is asking for this from ED unless they fix the replay feature. Which they should. But apparently this is a big deal. 

I gotta agree, if Replay is fundamentally broken, that need to be fixed first. As an intermediate solution: keep track and mission relatively static? 

BTW I asked ChatGPT 4 for help... 

How can I automatically start and stop capturing OpenXR performance when I replay a track in DCS?

To automatically start and stop capturing OpenXR performance when you replay a track in DCS, you can use a combination of scripting and OpenXR Developer Tools. Here's a step-by-step guide to achieving this:

1. Install Python: To run a custom script, you'll need Python installed on your computer. If you don't have Python, download and install it from the official website: https://www.python.org/downloads/
2. Install OpenXR Developer Tools: If you haven't already, download and install the OpenXR Developer Tools for Windows Mixed Reality from the Microsoft Store.
3. Create a Python script: Create a new text file and save it as "DCS_performance_capture.py" (or any name of your choice with the ".py" extension). Open the file in a text editor and add the following code:

import os
import time
import subprocess

# Configuration
dcs_executable = "Path\\to\\your\\DCS.exe"
track_file = "Path\\to\\your\\DCS\\Tracks\\your_track_file.trk"
openxr_dev_tools = "Path\\to\\OpenXRDeveloperToolsCLI.exe"
capture_duration = 120  # Adjust this value to match the duration of your track replay in seconds

# Start OpenXR Developer Tools performance capture
subprocess.Popen([openxr_dev_tools, "-CapturePerformanceData"])

# Start DCS with the track
subprocess.Popen([dcs_executable, "--track", track_file])

# Wait for the track to finish
time.sleep(capture_duration)

# Stop OpenXR Developer Tools performance capture
subprocess.Popen([openxr_dev_tools, "-StopPerformanceDataCapture"])

Replace the placeholders "Path\to\your\DCS.exe", "Path\to\your\DCS\Tracks\your_track_file.trk", and "Path\to\OpenXRDeveloperToolsCLI.exe" with the correct paths to your DCS executable, the track file you want to replay, and the OpenXR Developer Tools CLI executable, respectively.

Adjust the capture_duration variable to match the duration of your track replay in seconds.

4. Run the script: To start DCS, replay the track, and automatically capture OpenXR performance, simply run the "DCS_performance_capture.py" script. You can do this by double-clicking the script file or executing it from the command prompt:

python DCS_performance_capture.py

The script will start the OpenXR performance capture, launch DCS with the specified track, and stop the performance capture after the specified duration. Make sure to close any running instances of DCS before starting the script.

 

-------------------------

If this works then "HAIL CHATGPT" 

PS: I have zero Python experience. 

EDIT: Close but no cigar; see below. 

EDIT2: HAIL CHATGPT!! sort of...

Edited March 15, 2023 by Flextremes

[SharpeXB]

18 minutes ago, Flextremes said:

I gotta agree, if Replay is fundamentally broken, that need to be fixed first

Right. First order of business, fix the replay function. Aside from this, creating a standard track or tracks within the game for benchmarking purposes would be very easy. But apparently fixing the replay is a very complex task.

Does a VR track keep the head movement in the replay? TrackIR does. This is important in keeping the results consistent. 

Edited March 15, 2023 by SharpeXB

[Flextremes]

31 minutes ago, SharpeXB said:

Does a VR track keep the head movement in the replay? TrackIR does. This is important in keeping the results consistent. 

 

Agreed, in VR your headset movements will override the view in the replay. Would be nice to disable head-set spacial inputs during track replay.

If not possible, we could at least just fix the view in the track: put your headset on the table, press numpad 5, then start the replay track. This way tracks replay is consistent, which is key. 

Edited March 15, 2023 by Flextremes

[WipeUout]

40 minutes ago, SharpeXB said:

Right. First order of business, fix the replay function. Aside from this, creating a standard track or tracks within the game for benchmarking purposes would be very easy. But apparently fixing the replay is a very complex task.

It s a balancing act.  When the simulation spends time recording and saving input (head movement, keyboard, joystick, etc.) it takes resources away from the main task: Simulate!

The reason why tracks are not 100% accurate is probably because the capture frequency (or sampling rate) is not keeping up and you end up loosing input.  To solve this you need to bump up sampling rate and consequently you redirect resources that should simulate instead of doing telemetry.  So the other solution is to re-design the capture engine, lots of work.

Make a track, keep it short and that will do the job.  Just don't use the F-14, it is broken as far as track recording.

[Flextremes]

5 hours ago, Flextremes said:

I gotta agree, if Replay is fundamentally broken, that need to be fixed first. As an intermediate solution: keep track and mission relatively static? 

BTW I asked ChatGPT 4 for help... 

How can I automatically start and stop capturing OpenXR performance when I replay a track in DCS?

To automatically start and stop capturing OpenXR performance when you replay a track in DCS, you can use a combination of scripting and OpenXR Developer Tools. Here's a step-by-step guide to achieving this:

1. Install Python: To run a custom script, you'll need Python installed on your computer. If you don't have Python, download and install it from the official website: https://www.python.org/downloads/
2. Install OpenXR Developer Tools: If you haven't already, download and install the OpenXR Developer Tools for Windows Mixed Reality from the Microsoft Store.
3. Create a Python script: Create a new text file and save it as "DCS_performance_capture.py" (or any name of your choice with the ".py" extension). Open the file in a text editor and add the following code:

import os
import time
import subprocess

# Configuration
dcs_executable = "Path\\to\\your\\DCS.exe"
track_file = "Path\\to\\your\\DCS\\Tracks\\your_track_file.trk"
openxr_dev_tools = "Path\\to\\OpenXRDeveloperToolsCLI.exe"
capture_duration = 120  # Adjust this value to match the duration of your track replay in seconds

# Start OpenXR Developer Tools performance capture
subprocess.Popen([openxr_dev_tools, "-CapturePerformanceData"])

# Start DCS with the track
subprocess.Popen([dcs_executable, "--track", track_file])

# Wait for the track to finish
time.sleep(capture_duration)

# Stop OpenXR Developer Tools performance capture
subprocess.Popen([openxr_dev_tools, "-StopPerformanceDataCapture"])

Replace the placeholders "Path\to\your\DCS.exe", "Path\to\your\DCS\Tracks\your_track_file.trk", and "Path\to\OpenXRDeveloperToolsCLI.exe" with the correct paths to your DCS executable, the track file you want to replay, and the OpenXR Developer Tools CLI executable, respectively.

Adjust the capture_duration variable to match the duration of your track replay in seconds.

4. Run the script: To start DCS, replay the track, and automatically capture OpenXR performance, simply run the "DCS_performance_capture.py" script. You can do this by double-clicking the script file or executing it from the command prompt:

python DCS_performance_capture.py

The script will start the OpenXR performance capture, launch DCS with the specified track, and stop the performance capture after the specified duration. Make sure to close any running instances of DCS before starting the script.

 

-------------------------

If this works then "HAIL CHATGPT" 

PS: I have zero Python experience. 

 

Haha, ChatGPT always goes for python first. Creating a CMD script is so much simpler. So I asked my AI friend and he delivered.  

@echo off

set "dcs_executable=Path\to\your\DCS.exe"
set "track_file=Path\to\your\DCS\Tracks\your_track_file.trk"
set "openxr_dev_tools=Path\to\OpenXRDeveloperToolsCLI.exe"
set /a capture_duration=120

start "" "%openxr_dev_tools%" -CapturePerformanceData
start "" "%dcs_executable%" --track "%track_file%"

timeout /t %capture_duration%

start "" "%openxr_dev_tools%" -StopPerformanceDataCapture

Replace the placeholders "Path\to\your\DCS.exe", "Path\to\your\DCS\Tracks\your_track_file.trk", and "Path\to\OpenXRDeveloperToolsCLI.exe" with the correct paths to your DCS executable, the track file you want to replay, and the OpenXR Developer Tools CLI executable, respectively.

Adjust the capture_duration variable to match the duration of your track replay in seconds.

Run the batch file: To start DCS, replay the track, and automatically capture OpenXR performance, simply run the "DCS_performance_capture.bat" script. You can do this by double-clicking the script file

-------------------

Before you run the batch file: the trick is to lay your head-set horizontal and stable on the table and press Numpad 5 first to ensure replay consistency.

Lemme see if this works...

EDIT:

LOL: OpenXRDeveloperToolsCLI.exe is a hallucination of ChatGPT 4, it does not exist. This seemed indeed too easy

EDIT2: the --Track commandline option for dcs.exe is also pure fantasy by Chat GPT 4 it seems 

EDIT3: the --track does seem to work, but the track is only automatically loaded, not started, you need to choose "Fly" at the briefing screen.

There are too many caveats: logging is started when DCS.exe starts (off course): it should only start when the replay starts. Nice try, but we need some help from Dcs devs here.

EDIT4: I found a solution: 

 

Edited March 15, 2023 by Flextremes

[SharpeXB]

47 minutes ago, Flextremes said:

Agreed, in VR your headset movements will override the view in the replay. Would be nice to disable head-set spacial inputs during track replay.

If not possible, we could at least just fix the view in the track: put your headset on the table, press numpad 5, then start the replay track. This way tracks replay is consistent, which is key. 

 

Well I imagine the benchmark track would just have a fixed viewpoint, no need for it to move. I don’t suppose VR head movement can be turned off like head tracking? A moving view does make for a better more varied benchmark though. It just needs to be the same in every replay. 

25 minutes ago, WipeUout said:

It s a balancing act.  When the simulation spends time recording and saving input (head movement, keyboard, joystick, etc.) it takes resources away from the main task: Simulate!

The reason why tracks are not 100% accurate is probably because the capture frequency (or sampling rate) is not keeping up and you end up loosing input.  To solve this you need to bump up sampling rate and consequently you redirect resources that should simulate instead of doing telemetry.  So the other solution is to re-design the capture engine, lots of work.

Make a track, keep it short and that will do the job.  Just don't use the F-14, it is broken as far as track recording.

The head tracking is important to see for entertainment purposes and maybe for bug reporting and such if you want to see what the player saw.

Edited March 15, 2023 by SharpeXB

[Flextremes]

With the help of ChatGPT 4 for inspiration and Mr OpenXR toolkit himself I made this janky but functional 'proof-of-concept' to consistently capture openXR performance data during track replay.

This is an AutoHotkey V1 script. Download from https://www.autohotkey.com/. 

Open notepad and save as, select "all file types" and name the file "BenchmarkDCS.ahk"

; manually Start DCS with the track

msgBoxShown := 0
MsgBox, 1 --- DO NOT CLICK [OK] BEFORE COMPLETING THESE ACTIONS ---  `n1 Place your VR headset perfectly horizontal on a table. `n2 Select the DCS desktop mirror.`n3 Load the track in DCS you want to replay. `n4 Do NOT select FLY at the briefing screen YET! `n4 Select numpad 5 to center the view. `n5 Only now you can close this messagebox `n6 ALT-TAB to DCS briefing screen and use the Pause Key to start replay. `n7 OpenXR stats recording will autostart and stop after 10 seconds.
msgBoxShown := 1

~Pause::
 if (msgBoxShown = 1) {
Run, `"C:\Program Files\OpenXR-Toolkit\companion.exe`" -record-stats toggle
Sleep 10000
Run, `"C:\Program Files\OpenXR-Toolkit\companion.exe`" -record-stats toggle
msgbox, Finished! Open %localappdata%\OpenXR-Toolkit\stats for the stats CSV file
Run, `"%localappdata%\OpenXR-Toolkit\stats`"
exitapp
    }
return

Please note: you need to specify the time in ms at the SLEEP function for how long you want to record data. 10000 is 10 seconds, which is used in this example.

To run a benchmark:

1. Just load DCS in VR
2. put your headset perfectly horizontal on your desk
3. goto the DCS desktop mirror
4. select numpad 5
5. select your track in replay
6. Do NOT select FLY yet...
7. Right click BenchmarkDCS.ahk script below and start with AutoHotkey launcher
8. When ready: press OK in the messagebox.
9. ALT-TAB to the briefing screen in the DCS Desktop Mirror 
10. Press the PAUSE Key to start the replay (do NOT mouse click on 'Fly', press the Pause key instead)

After you close the messagebox the script will wait for a Pause Key Press. When the Pause is pressed the replay is started, and OpenXR performance statistics are automatically started at the exact same time by this script

After 10 seconds (this is configurable at the Sleep function in the script) you can review the CSV file in %localappdata%\OpenXR-Toolkit\stats.

Have fun benchmarking :))

 

Edited March 16, 2023 by Flextremes

[Pikey]

The above benchmark steps can be achieved with a ten year old tool FRAPS, with its Benchmark tool ticked, and FRAFS Bench Viewer, that coverts the csv to a graph instantly, both freeware, example below. Unfortunately tracks are misunderstood. Tracks will mimic a single player or server host experience which isn't the entire story.

Benchmarking internally via a specific in game tool was an old thing. Back in the days of the VOODOO 2 Extreme graphics cards, when we marvelled at textures being on polygons at the same time!! Static benchmarks are not useful for CPU centric applications like DCS. The additional CPU stress is caused by AI with realtime decision making, Taxiing AI proximity checks, hundreds of units performing realtime detection checks on every sensor type they have, each. Ground units with realtime shortest path problem solving algorithms, Missiles with realtime guidance. Tracks are actually very good for reproducing exactly that noise overhead, a single plane does surprisingly little on its own. And once you deny the system CPU, your frames can be measured properly. Of course, your client doesn't do half of those things when on line, so you can throw this all out when considering offline benchmarking. Additionally, with the rendering thread now broken off, we are in a new era.

Given that the difference between Caucasus Anapa and Mount Pleasant, Falklands can be measured in VR in double digits, my recommednation is that for play, you examine your worst case scenario and tune from there by feeling rather than counting the milliseconds, because the amount of variables in consideration stretches beyond any simple benchmark.

 

[Flextremes]

4 hours ago, Pikey said:

The above benchmark steps can be achieved with a ten year old tool FRAPS, with its Benchmark tool ticked, and FRAFS Bench Viewer, that coverts the csv to a graph instantly, both freeware, example below. Unfortunately tracks are misunderstood. Tracks will mimic a single player or server host experience which isn't the entire story.

Benchmarking internally via a specific in game tool was an old thing. Back in the days of the VOODOO 2 Extreme graphics cards, when we marvelled at textures being on polygons at the same time!! Static benchmarks are not useful for CPU centric applications like DCS. The additional CPU stress is caused by AI with realtime decision making, Taxiing AI proximity checks, hundreds of units performing realtime detection checks on every sensor type they have, each. Ground units with realtime shortest path problem solving algorithms, Missiles with realtime guidance. Tracks are actually very good for reproducing exactly that noise overhead, a single plane does surprisingly little on its own. And once you deny the system CPU, your frames can be measured properly. Of course, your client doesn't do half of those things when on line, so you can throw this all out when considering offline benchmarking. Additionally, with the rendering thread now broken off, we are in a new era.

Given that the difference between Caucasus Anapa and Mount Pleasant, Falklands can be measured in VR in double digits, my recommednation is that for play, you examine your worst case scenario and tune from there by feeling rather than counting the milliseconds, because the amount of variables in consideration stretches beyond any simple benchmark.

 

Appreciate your feedback, but I do not completely understand your point.

Isnt Fraps is seriously outdated, only provides fps statistics (and not frame times which is crucial to understand VR performance, reported by openxr)? Is it not better to recommend a more up-to-date generic/3rdp-party performance analysis tools like Nvidia Frameview?

More importantly, Fraps does not provide a standardized way to measure performance metrics.  You need to manually start and stop fraps which makes this method everything but exact. How can you compare your results with that of others?

Not sure I follow your comment on in-game benchmarks: so many modern games and AAA titles (from GTA5 to Cyberpunk 2077) have in-game benchmarking option, it is extremely common still nowadays.

Edited April 7, 2023 by Flextremes

[Pikey]

Well there we go, two people two answers. That's where we leave things.