VR benchmarking methods

[Fakum]

6 hours ago, Qcumber said:

Looking at your signature it looks like you are using a monitor rather than VR so measuring FPS and 1% lows is the right approach. With VR the maximum FPS is locked by the headset refresh rate so looking at 1% lows does not necessarily give you the right information when comparing different settings and situations. 

@QcumberYou are correct, I am on a monitor at this time, I am awaiting receipt of the Pimax Super. While I am waiting, I am tuning Windows 10 and doing as much preliminary stuff as I can so once received, I can setup and get VR dialed in. Part of that is this. So are you suggesting that benchmarking the VR Headset in this manner is not the way to go? I would think comparing Average FPS and 1% Lows in either monitor or VR would be beneficial. Thanks for your response.

[Qcumber]

10 minutes ago, Fakum said:

@QcumberYou are correct, I am on a monitor at this time, I am awaiting receipt of the Pimax Super. While I am waiting, I am tuning Windows 10 and doing as much preliminary stuff as I can so once received, I can setup and get VR dialed in. Part of that is this. So are you suggesting that benchmarking the VR Headset in this manner is not the way to go? I would think comparing Average FPS and 1% Lows in either monitor or VR would be beneficial. Thanks for your response.

I am in the process of updating my spreadsheet to measure 1% and 0.1% lows, as well as other measures, so it might help you once you get your Pimax Super. 

[Fakum]

@Qcumber Thanks, I appreciate your response and your efforts. I'm not sure that updating your spreadsheet will help me to understand your chart, but like I said, I believe what I have accomplished so far via MSI and getting the log that indicates the AVG FPS along with the 1% Lows satisfies my quest (although not represented graphically). My point is, that your response to me having a monitor at this time left me to believe that I was not doing what would be of any benefit when it comes time to running the same benchmarks using VR. That is what i wanted to clear up. If what I have setup now is not the way to proceed when trying to dial in performance of VR, now is the time for me to resolve that. Thanks,

[whitav8]

The spreadsheets do not expect the third column in the CSV file called "Display XrTime" (added by the v020)  so you can either delete that column in the CSV file or add a column to the T1, T2,...Data copy area.

At least that was something I had to deal with.

 

[Qcumber]

5 hours ago, whitav8 said:

The spreadsheets do not expect the third column in the CSV file called "Display XrTime" (added by the v020)  so you can either delete that column in the CSV file or add a column to the T1, T2,...Data copy area.

At least that was something I had to deal with.

 

Thanks. I'll update the spreadsheet. 

[Qcumber]

I have developed this analysis a bit more. I will update the first post soon and add my new spreadsheet but just wanted to share some of what I have added and why it might be useful.

This will focus on FPS and a slightly different way of looking at it. The table below shows the FPS values for three different tests with different loads on the CPU and GPU. 

• Test 1 is an F-4 take off from Syria (heavy)
• Test 2 is an F-16 flight over Syria (medium)
• Test 3 is a BF-109 flight over the desert on the Sinai map (low)

FPS mean, 1% and 0.1% lows should be obvious. I have also added a "smoothness" score which predicts the range of variation from the average FPS value*. This should be more accurate than 1%/0.1% lows as it also includes values above the mean FPS and the full range in-between. As general rule with a smoothness score of below 90% you are likely to see some stutters, becoming more pronounced the lower the value. Values above 90% should have increasing perceptions of smoothness.

Here is a plot of the distribution of the FPS count displayed as a frequency of occurrence. Basically the taller and narrower the shape of the track, the smoother it will look as most of the frame intervals fall around the 72 fps mark. As you can see in the FPS distribution chart (top) test 1 is much broader and flatter than tests 2 or 3, even though all three can maintain an average FPS of 72. This difference is represented in smoothness scores of 81%, 93% and 95% respectively.

 

 

* For anyone interested in statistics I calculated "smoothness" as [1-Coefficient of Variation]x100. Credit to "tundrowalker" for the idea (https://www.moddb.com/tutorials/measuring-fps-performance-beyond-lows).

Edited May 4 by Qcumber

[whitav8]

A measure that I think could be really important is the number of clearly detectable visual "stutters" per 10 seconds or so ( just a quick example ). This would be like Qcumber's smoothness but would try to actually find in the data a visually detectable stutter. Surely the really long frames maybe 3 ( or so) msec greater than the average are stutters but also it might be the difference in msec. from one frame to another or some algorithm. It's not clear to me just what we should look for. When I try to use SSW on the Quest3 in "automatic" mode, I think it looks worse because it shifts from 36FPS to 72FPS frequently. I would rather disable SSW but try to choose DCS settings that keep this "stutter rate" below a certain value. I wish I could actually control the generation of a stutter so I could test to see how to measure from XRFrameTools data. Also, I do expect DCS itself with changing scenery chunks or maybe turn rate in a dogfight to generate stutters but what I want to dig out are more like stutters that are caused by exceeding the graphics "headroom". A challenging subject. 

 

[Qcumber]

1 minute ago, whitav8 said:

A measure that I think could be really important is the number of clearly detectable visual "stutters" per 10 seconds or so ( just a quick example ). This would be like Qcumber's smoothness but would try to actually find in the data a visually detectable stutter. Surely the really long frames maybe 3 ( or so) msec greater than the average are stutters but also it might be the difference in msec. from one frame to another or some algorithm. It's not clear to me just what we should look for. When I try to use SSW on the Quest3 in "automatic" mode, I think it looks worse because it shifts from 36FPS to 72FPS frequently. I would rather disable SSW but try to choose DCS settings that keep this "stutter rate" below a certain value. I wish I could actually control the generation of a stutter so I could test to see how to measure from XRFrameTools data. Also, I do expect DCS itself with changing scenery chunks or maybe turn rate in a dogfight to generate stutters but what I want to dig out are more like stutters that are caused by exceeding the graphics "headroom". A challenging subject. 

 

It should be possible to look for variations in fps per unit time. I would need to think about how to do it though. 

[Ready]

Nice! Have you considered/looked into CapFrameX? I used to perform the Excel calculations before as well using various tools to record, export and analyze. I found that https://www.capframex.com/ is way easier to quickly get to useable visual data and to quickly compare settings. No more exporting, importing, manual stuff, no more Excel wizardry needed. Just run the game and use hotkeys to start/end the log recordings, add titles/descriptions to the recording tracks and voila. I think it will save you a lot of time going forward and I assume it will have most of what you are looking for. I have used it 500+ times in my quest to silence Windows, stabilize my rig and get the most performance out of the hardware, testing all setttings and scenario's I could imagine and stuff I found online, I still use it to test the effects of updates or setting changes. You can find some decent examples of this tool in the postings of my thread (don't want to self plug). I also use XRFrameTools, but mostly to get realtime visualization of stability and get direct feedback of things I change on the fly. I now only use Excel to keep track of all the changes I make (either by for example DCS updates, driver updates or me tweaking something); with each change I create a log entry and log the frametimes etc. This ends up in one chart to show historically how my frametimes are improving or degrading with changes. It also is great to lookup stuff I have tested. Hope it is of any help!

 

 

 

Edited June 1 by Ready

[Qcumber]

10 minutes ago, Ready said:

Nice! Have you considered/looked into CapFrameX? I used to perform the Excel calculations before as well using various tools to record, export and analyze. I found that https://www.capframex.com/ is way easier to quickly get to useable visual data and to quickly compare settings. No more exporting, importing, manual stuff, no more excel. Just run the game and use hotkeys to start/end the log recordings, add titles/descriptions to the recording tracks and voila. I think it will save you a lot of time going forward and I assume it will have most of what you are looking for. I have used it literally hundreds of times in my quest to silence Windows, optimize and stabilize my rig, testing all setttings and scenario's I could imagine and stuff I found online, I still use it to test the effects of updates or setting changes. You can find some decent examples of this tool in the postings of my thread (don't want to self plug). I also use XRFrameTools, but mostly to get realtime visualization of stability and get direct feedback of things I change on the fly. I now only use Excel to keep track of all the changes I make (either by for example DCS updates, driver updates or me tweaking something); with each change I create a log entry and log the frametimes etc. This ends up in one chart to show historically how my frametimes are improving or degrading with changes. It also is great to lookup stuff I have tested in the past to compare. Hope it is of any help!

 

 

 

Thanks. I'll take a look. The main reason for doing it this way is because XRFrameTools is optimized for VR. I did enjoy the process of using excel but now I have less time so anything that could automate the process would be good. 

[actually_fred]

For openxr, capframex, RTSS, presentmon and others may be easier, but the numbers they give are wrong, and they can actively harm the VR experience because they work on the “send to mirror window” part of the game, not the VR part.

https://github.com/fredemmott/xrframetools?tab=readme-ov-file#why-should-i-use-this-instead-of-my-favorite-tool-for-non-vr-games

If you go beyond monitoring with these into framerate limiting, vsync, or other modifications, these will either do nothing to VR beyond placebo, or tie your headset to your monitor's timing rather than the runtime's timing, often leading to stutters, motion misprediction, and other tracking issues.

Edit: for example, accurate support for OpenXR has been an open feature request for CapFrameX for over a year: https://github.com/CXWorld/CapFrameX/issues/277

While Google's AI and a guest post on Pimax's blog say PresentMon support OpenXR, this is incorrect. This is obvious from the fact that it does not have an API layer. The only other way it could provide accurate numbers would be if it integrated with the runtimes - given PresentMon itself is open source, it would be an unusual choice for them to integrate with some runtimes, but none of the open source ones (like VDXR) - and another bad choice to display incorrect information on other runtimes rather than a 'runtime not supported' message which woudl require an API layer. PresentMon can hint about your load, but this says nothing about timings/bottlenecks unless something is at 100%. I'm also not seeing any trace of OpenXR-related code in PresentMon's source code.

Edited June 2 by actually_fred

[Qcumber]

Spreadsheet and original post updated (v4).

• Updated CPU latency to include both app and render time.
• Default latency setting on charts set to 30 ms
• Added FPS by occurrence and Frame Interval.
• Added new feature called "smoothness" which calculates the variation of FPS and Frame Interval from the mean. This is useful if the average FPS counter shows that you have 72 FPS but you still may have some stutters. 

[Qcumber]

Call for data

If anyone is interested please could you use the guide in this thread to record tracks with various parameters and then send them to me so that I can collate and analyse data from different setups.

I am interested in the following at first:

• DLAA/DLAA vs MSAA and no AA.
• Effects of QVFR
• Visibility range.
• Pre-load radius.
• Detail sliders.
• Shadow detail.

If possible please send me recordings of the same track with only one parameter changed. 

I will need the following data alongside the XRFrametools log file: 

• PC specs
• VR headset: including method of connection (e.g. Meta link, Virtual Desktop etc.), base resolution and any scaling.
• DCS settings
• Use of any other other third party software and/or XR layers (e.g. OXRTK, XRNecksafer etc.)
• If using QVFR the config settings. 

NB: Please remember to disable "Turbo mode" or "prefer framerate over latency" as these will invalidate the measurements. 

 

[Qcumber]

On 5/4/2025 at 9:21 PM, whitav8 said:

A measure that I think could be really important is the number of clearly detectable visual "stutters" per 10 seconds or so ( just a quick example )

Sorry but I have only just getting around to looking at this. Life has been busy.

To clarify, you are suggesting that we look at the number of events where the Frame Interval exceeds that required to maintain the target frame rate. But we would also need to know the relative "size" of the deficit. So we would need to look at the number which exceeds the Frame Interval by levels. For example, if the target Frame Interval for 72 fps is 13.8 ms then we would need to look at perhaps 1 ms bins (13.9-14.8, 14.9-15.8, 15.9-16.8 etc.).

Have I got this right? I will have a think about how I could do this and what information it would give us and how to display it in a format that makes sense. 

[Qcumber]

@whitav8 I have updated my working spreadsheet to include levels of FPS as a factor of refresh rate times 0.95, 0.75, 0.5 and 0.33 (at 72Hz this would be 68, 54, 36 and 24 fps). However, I still think that measuring coefficient of variation is better as this looks at the standard variation in fps relative to the mean fps, including fps going higher than 72 fps. Remember that the fps here is calculated from Frame Interval for each frame. This is not in version 4 as I still need to work on this. 

For the examples below 20% of frames drop below the Frame Interval needed for 68 fps and only 0.4% drop below the frame Interval Needed for 54 fps. 

[Fakum]

Thanks for all of your work sir! I will dabble into this again to check it out. I love the addition of the "Smoothness", after all, that is the bottom line right, at least that is what I try to target, not chasing high frame rates, but avoiding stutters.