VR benchmarking methods

[Qcumber]

For those of you who have not seen this yet (and at the request of a couple of forum users) here is a method I have been using to benchmark VR.*

Please note that you will need to have some experience with using spreadsheets before trying this. I do plan on making the workflow a bit more automated if I can but have not yet got round to this. Any advice would be welcome. 

I have attached my blank spreadsheet below.

For this you will need to download the software XRFrametools by Fred Emmott.

Release v0.2.0: bugfixes · fredemmott/XRFrameTools · GitHub. The latest version has resolved the previous issues so it should work OK for data collection and export. 

[please note there is currently an issue with converting the log file to a CSV file so you need to use another version for this purpose. You will need to download this here https://github.com/fredemmott/XRFrameTools/actions/runs/13800544875/artifacts/2734110955 (I'll call this version 2 so that it is a bit clearer later). Fred has said that this will be consolidated in the next update])

This will allow you to capture individual frame measurements within DCS. The ones we are interested in primarily are CPU render time, GPU render time, VRAM use and FPS.

Please let me know if this is useful as I would like to see how it compares with other setups.

Steps:

1. Open DCS in VR and load a mission that you want to use as a test. I would advise recording a 60s track for reproducibility. Get everything ready then hit pause and Alt-tab out of DCS. Make sure that Turbo mode is disabled in OXRTK or QVFR if you are using it.
2. Run XRFrameTools and navigate to the "Perfromance Logging" tab. Select "Enable for>1 minute". Then Alt-tab back to DCS and un-pause. Let the track run for 60s then exit. 
3. Open "Performance log>convert log to CSV" then adjust the setting for "1 frame per CSV row". Click save.
4. Open the CSV file and then copy the contents into one of the test pages in the attached spreadsheet (T1, T2 etc.). These should be copied into columns P:AL. Please note that the number of rows can vary a little so check at the bottom of columns E:H and delete any rows with "0" if you want correct mean, min, max etc. . 
5. You can then view the charts in the first tab of the spreadsheet along with mean data in the table. 

Example

This example compares DLSS preset "K" (T1) with "J" (T2). I have also included this as a spreadsheet so that you can see my workings.

The data here is a plot of the frequency at which GPU render times occur (as a percentage). The highest frequencies occur around the 8-9 ms mark as per the highest "peak" with a minimum of 7.13 ms and a max of 14-15 ms (although there are some outliers as high as 20 ms).

In this case preset "K" has a slight edge over preset "J" with a mean latency difference of about 0.5 ms with slightly less "missed frames"; those which occur beyond the 13.88 ms red line which is the minimum latency needed for 72fps. Anything beyond this can result in missed frames and a stutter. The difference is not really significant though. I would need to run this a few times and take an average. 

*This is based on the method used in this YouTube channel "Benchmark Odyssey". So thanks for the ideas an inspiration

UPDATE 13 April 2025 (v2)

The spreadsheet has been updated as follows:

• Maximum latency now 50 ms for CPU and GPU charts
• FPS count CPU render and GPU render over time added.
• Vertical lines added to represent 90, 72, 45 and 36 FPS respectively (the last 2 relate to reprojection at 90Hz and 72Hz).

UPDATE 28 April 2025 (v3)

The spreadsheet has been updated as follows: 

• Fixed some errors with formulas.
• Added VRAM chart.
• Reduced maximum chart latency to 30 ms (if anyone wants to extend this then you can just change this in the chart data). 

 

 

 

XRFrametools Benchmark blank (V3) forum copy.xlsx

Edited April 28 by Qcumber

[Qcumber]

An alternative method for capturing the data is to use the OXRTK "Record statistics" function. This is at the bottom of the first tab. Enable "High Rate Statistics". 

The log file is created here:

%LocalAppData%\OpenXR-Toolkit\stats

This is an easier method but only gives you an average of about 7-8 frames at 72Hz. The spreadsheet will need adjusting accordingly using this method. 

Overall I prefer the XRFrameTools method as it captures every frame. 

[whitav8]

Sorry but I can't access the "version 2" of XRFrameTools- is that maybe an older version 0.4 which I did find.

[Qcumber]

52 minutes ago, whitav8 said:

Sorry but I can't access the "version 2" of XRFrameTools- is that maybe an older version 0.4 which I did find.

You need to get a direct download. Here is the thread I started with Fred on GitHub. Hopefully that link will work ok. Just use this version for the log conversion not the capture. 

https://github.com/fredemmott/XRFrameTools/issues/34#issuecomment-2737326559

XRFrameTools is still in "alpha" but hopefully there will be an update soon so that this issue is fixed. 

[coldViPer]

Hmm why I have only CPU curve but no GPU? Attached my examples, what I'm missing?

XRFrametools Benchmark blank.xlsx DCS 2025-04-12 15-00-35.0605237 UTC.csv

[Qcumber]

6 minutes ago, coldViPer said:

Hmm why I have only CPU curve but no GPU? Attached my examples, what I'm missing?

XRFrametools Benchmark blank.xlsx 3.11 MB · 1 download DCS 2025-04-12 15-00-35.0605237 UTC.csv 466.49 kB · 0 downloads

The GPU frame times are longer than 20 ms so they were not showing. I have extended the chart to 30ms. I've updated the SS for you. What DCS settings are you using and what is the module/map/track? 

 

ColdViPer benchmark.xlsx

[coldViPer]

Oh thanks. I'm using 60 second track (Kola / F-18 / flying low over Rovaniemi AB), DCS settings quite maxed out.

Kola_bencmark.trk

[coldViPer]

13 hours ago, Qcumber said:

The GPU frame times are longer than 20 ms so they were not showing. I have extended the chart to 30ms.

 

Can I edit this chart, how do I extend this GPU frame time to 50ms? Just testing one thing.

[Qcumber]

11 minutes ago, coldViPer said:

Can I edit this chart, how do I extend this GPU frame time to 50ms? Just testing one thing.

Yes. Just go into the chart and increase the data range for that sample. I think I have included up to 50ms. If no then I can always update the blank spreadsheet and repost it. 

[coldViPer]

2 hours ago, Qcumber said:

Yes. Just go into the chart and increase the data range for that sample. I think I have included up to 50ms. If no then I can always update the blank spreadsheet and repost it. 

Well if you have time to update the blank sheet to 40ms and share it i'd be grateful. I found that data range and tried to change values but it messes up the view totally for some reason (Office 365).

[Qcumber]

59 minutes ago, coldViPer said:

Well if you have time to update the blank sheet to 40ms and share it i'd be grateful. I found that data range and tried to change values but it messes up the view totally for some reason (Office 365).

I'll update it later so that it has up to 50 ms then set all the charts to the max value. Then it will be standardized for reprojection frametimes too. I probably won't get chance until this evening UK time though. 

[Qcumber]

UPDATE 13 April 2025

The spreadsheet has been updated as follows:

• Maximum latency now 50 ms for CPU and GPU charts'
• FPS count CPU render and GPU render over time added.
• Vertical lines added to represent 90, 72, 45 and 36 FPS respectively (the last 2 relate to reprojection at 90Hz and 72Hz).

[The updated file is in the first post]

The charts now look like this:

[coldViPer]

Nice, thanks!

[coldViPer]

Can anyone explain why I get much accurate data when "prefer framerate over latency" is ON in OpenXR Tools? I assume this is like TURBO mode ON,  I thought this setting gives wrong data when enabled. In the file T1=ON, T4=OFF.

XRFrametools Benchmark TURBO compare.xlsx

[Qcumber]

1 hour ago, coldViPer said:

Can anyone explain why I get much accurate data when "prefer framerate over latency" is ON in OpenXR Tools? I assume this is like TURBO mode ON,  I thought this setting gives wrong data when enabled. In the file T1=ON, T4=OFF.

XRFrametools Benchmark TURBO compare.xlsx 2.99 MB · 0 downloads

Interesting results. That is a huge difference in GPU latency. It might be an error in the capture. If I enable Turbo mode it gives weird results:

https://github.com/fredemmott/XRFrameTools?tab=readme-ov-file#supported-measurements

 

 

[Qcumber]

Sorry. Just looking at this again. I had misunderstood. Yes, it does look like the more accurate results are with prefer "framerate over latency" is ON. I presume you are running at 90Hz with reprojection? It would worth repeating to see if the results are consistent. 

Is this a very demanding track? It looks like you have a lot of missed frames even with reprojection. Do you get any stutter? 

[coldViPer]

On 4/13/2025 at 11:58 AM, Qcumber said:

 

This is quite demanding track with high dcs settings, the same Kola track I attached earlier. I don't have any reprojection enabled. Normally I use Rivatuner to lock 45fps but always disable this when benchmarking like now. Yes its quite stuttery when flying without Rivatuner 45 fps lock but very smooth when enabling it.

[Qcumber]

I have updated the first page to reflect an update of XRFrametools and a newer version of the Spreadsheet. 

[Fakum]

I am trying to understand how to read these graphs. I simply want to monitor FPS and Frame Times to look at FPS & 1%. For example, I understand this chart:

null

 

But I dont really understand this chart:

null

It looks like it starts out at 20ms, ends at 26 ms, but what is the freq % mean?

 

 

[Qcumber]

6 hours ago, Fakum said:

I am trying to understand how to read these graphs. I simply want to monitor FPS and Frame Times to look at FPS & 1%. For example, I understand this chart:

null

 

But I dont really understand this chart:

null

It looks like it starts out at 20ms, ends at 26 ms, but what is the freq % mean?

 

 

The frequency is the number of frames that fall within 0.1 ms latency periods. XRFrametools captures individual render times. The spreadsheet looks for events that fall within 0.1 ms periods and counts them. This is then expressed as a percentage of the total number of events. The shortest latencies are being picked up at around 20 ms, peaking at 21-23 ms and extending to 26 ms. You have not included the other charts or table but I would estimate that FPS is 37-50 with mean around 45. FPS is [1000/GPU-render-time] so you need a minimum of 27.8 ms to maintain 36 FPS. However, CPU render time will impact in this if it is too long and cannot deliver to the GPU on time. What do the CPU render times look like? Can you show the FPS chart? This will show the variance and spikes. 

[Fakum]

Thanks for the response and explanation. That is NOT my chart, its a clip I took from your post in this thread, just to use as an example to make an example of what I did not understand. As much as I appreciate your effort to explain the graph, I still dont get it, and thats on me. I realize the objective would be to get performance dialed in to maintain consistency with frame times instead of having frequent spikes in ms, which cause the stuttering. Im just trying to find a way to graph that. Like I said, I understand this chart, and I can see how useful it would be to make comparisons between alterations.

null

[Fakum]

Since I didnt get my head wrapped around that XR Frametools graph, I setup MSI Afterburner and the Log I created from a 32 second benchmark gives me the values of what i was looking for. Albeit not an actual chart, its the two numbers I want to see when making comparisons with other benchmarks I will do after making changes. Am I on the right track here? You want the 1% low framerate to be as close to the Average framerate for best, smooth performance right? Thanks.

null

Edited May 1 by Fakum

[Qcumber]

5 hours ago, Fakum said:

Since I didnt get my head wrapped around that XR Frametools graph, I setup MSI Afterburner and the Log I created from a 32 second benchmark gives me the values of what i was looking for. Albeit not an actual chart, its the two numbers I want to see when making comparisons with other benchmarks I will do after making changes. Am I on the right track here? You want the 1% low framerate to be as close to the Average framerate for best, smooth performance right? Thanks.

null

Looking at your signature it looks like you are using a monitor rather than VR so measuring FPS and 1% lows is the right approach. With VR the maximum FPS is locked by the headset refresh rate so looking at 1% lows does not necessarily give you the right information when comparing different settings and situations. 

[coldViPer]

I don't understand, why XRFrameTools v020 gives me weird CPU / GPU data. When tried old v010, that seems to be normal. In attached file, T1 is new version 020, T4 is old 010

XRFrametools Benchmark blank (V3).xlsx

[Qcumber]

1 hour ago, coldViPer said:

I don't understand, why XRFrameTools v020 gives me weird CPU / GPU data. When tried old v010, that seems to be normal. In attached file, T1 is new version 020, T4 is old 010

XRFrametools Benchmark blank (V3).xlsx 3.6 MB · 0 downloads

Something strange going on there.

I've not been able to try it yet as I've been away. I'm back tomorrow so will give it a go.