Comparative benchmarking

Since we thoroughly derailed Wags' thread on his new computer purchase, thought we might as well run some comparative tests to see if we can put to bed the whole debate about which processors run how well. To that end I've run Instant Action, and then loaded the track (attached) to let it play while FRAPS ran a 10-minute FPS count.

During testing my settings were thus:

Standard "high" setting, plus cockpit displays at 1024, mirrors on, 4xAA and 8xAF. I didn't go higher on the AA and AF to ensure that it's closer to where most people can participate and measure something that for them would be relatively close to what they run when they play normally. I realize some people (I hate you Viper) will run higher than this, and tests on higher settings wouldn't hurt.

Computer setup for my test was a 1680x1050 screen, Intel e8500 running at 4GHz, DDR2 RAM running at 992MHz ( 5-5-5-18 ), and a GeForce 9800GTX+ 512MB running at stock speed. Previous tests over at Wags' thread seemed to indicate that RAM wasn't much of an issue - at least not in the non-combat track that was used there.

This track includes wingman action (and death :D ), cannon employment, rockets, vikhrs, lots of scanning with the Skhval and flares.

Results:

Min: 38

Max: 81

Avg: 52,577

I do have the raw data as well on hand, but don't have time to make graphs of it right now. :P

instant.trk

Track as above:

Asus P5Q-E

E8500 @ 3.8GHz

2x 2GB Corsair DDR2 RAM @ 533MHz 5-5-5-18

Sapphire HD4890

1680x1050, 4xAA, 8xAF, Default High settings

Mirrors ON, Cockpit 1024-every frame

Windows XP Pro SP3

Min: 18

Max: 61

Avg: 38.527

Windows 7 64bit

Min: 24

Max: 71

Avg: 43.078

Mirrors OFF, Cockpit 1024-every frame

Windows XP Pro SP3

Min: 29

Max: 80

Avg: 43.227

Windows 7 64bit

Min: 26

Max: 88

Avg: 47.896

Mirrors ON, Cockpit 1024

Windows XP Pro SP3

Min: 32

Max: 70

Avg: 44.297

Windows 7 64bit

Min: 32

Max: 78

Avg: 49.397

And if I set my native resolution to 1920x1200

Mirrors OFF, Cockpit 1024, Water=1

Windows XP Pro SP3

Min: 35

Max: 85

Avg: 48.548

Windows 7 64bit

Min: 35

Max: 95

Avg: 53.717

I am very surprised that difference between Cockpit Resolution 1024 and 1024-every frame is so different and also Windows 7 64bit is in general some 10% faster then XP

Track as above:

Asus Rampage Extreme

E8600 @ 3.8 GHz

4GB DDR3 Kingston RAM

Asus Radeon HD 4870x2 2GB GDDR5

#1

Settings

1920x1200

Default High, Cockpit @ 1024, Mirrors On;

4xAA, 8xAF:

Results

Min: 27

Max: 67

Average: 41

#2

Settings

1920x1200

Default High, Cockpit @ 1024, Mirrors Off;

4xAA, 8xAF:

Results

Min: 29

Max: 72

Average: 42

#3 ( Personal Preferred Gameplay setting)

Settings

1920x1200

Default High, with Civilian Traffic On/Water Low;

Cockpit @ 512 (No Visual Difference personally from 1024), Mirrors Off, Shadows All Planar;

32xAA, 16xAF:

Results

Min: 30

Max: 88

Average: 45

Kuky, that is a very interesting observation. I'll have to re-do my own benchmark to check that out as well.

Also, I would LOVE a 4GHz AMD system to report it's findings, and a 4GHz i7, since we have only variations on the C2D so far. Previous tests seemed to indicate that memory was not really much of an issue so the DDR2/DDR3 thing shouldn't skew the data too much.

First off here is my computer:

Windows 7 Professional 64bit

Core 2 Duo E6750 2.66Ghz overclocked to 3.4Ghz (~28% overclock)

4GB DDR2 with a slight (1:1 ratio) overclock

GeForce 8800GTS 640MB (G80) @ 650Mhz core / 1000Mhz memory (~25% overclock)

-------

Here are my results from one run with my preferred settings:

Textures HIGH

Scenes HIGH

Civ Traffic YES

Water OFF (DISABLED IN LUA (0))

Visib Range HIGH

Heat Blur ON

Shadows ACTIVE PLANAR

Resolution 1680x1050

Aspect 16:10

Monitors 1 SCREEN

Res. of Cockpit Displays 256

Mirrors OFF

Vertical Sync ON (nVidia ControlPanel)

Anisotropic Filtering FORCED 16x (nVidia ControlPanel)

Anti-Aliasing FORCED 4x (nVidia ControlPanel)

Results:

Frames: 62010

Time: 1191452

Min FPS: 12

Max FPS: 62

Avg. FPS: 52.046

Comments:

The FPS felt solid and the gameplay fluid until the obsessive spamming of flares(!) when the FPS dropped to the low-point of 12. Looking into the smoke stack left by the dying flares is a real fps-killer for me =).

I've attached a plot of the FPS vs time in case anyone is interested in comparing. The blue line is FPS at each time unit and the black line represents a moving average with a period of 60 readings. I'm not sure at what interval FRAPS stores the FPS-data so the scale of the time-axis is somewhat strange. Just consider it as 1=start and 1192=end =D

-----------

I'm planning on adding results from runs with default settings HIGH and MEDIUM as well in case there is some prospective buyer finding this thread in the future with a setup as old as mine =) I'll add them here when I've got the time.

Aye, I did the flare spam as a possible position to test their impact. I'll do a run through the entire track at some point and see what impact they have on my hardware. Also on my to-do list is some comparisons where I lower my processor clock to see if it scales directly to processor speed.

Actually I did some primitive benchmarking in the early days of Black Shark and it's available in this thread:

http://forums.eagle.ru/showthread.php?t=36500

specifically these two posts:

http://forums.eagle.ru/showpost.php?p=613309&postcount=37

http://forums.eagle.ru/showpost.php?p=613550&postcount=40

Obviously it isn't the best test setup but it kind of shows that performance is very cpu-clock-dependant. Looking forward to your results!

My wet dream would actually be a C2D or C2Q with an unlocked multiplier to investigate the impact of the FSB. As it is with my locked C2D it's impossible to differentiate, and while it's fair to think that it is the CPU it would be interesting to see if we could find a situation where the bus becomes the bottleneck.

I'll get a few tests running this evening.

EDIT: Also, it is encouraging that no-one has offered harsh comments about my flying skills yet. :D

EDIT2: Actually, I'm a moron. It IS possible, even easy, to test FSB impact. The C2D is only locked upwards, so if there is negligible difference between clock speed X achieved through lower multiplier but high FSB and the opposite... We'd know. I shall add that to my list of tests I need to run. :D

My new(ish) rig:

Asus P6T

Intel i7 920 @ 4GHz

3x 2GB OCZ DDR3 PC3-12800 @ 800MHz 8-8-8-24

ATI Sapphire HD5850 @ 950 - 1300

Windows 7 Evaluation copy 7600 64bit

1680x1050, 4xAA (box, multisample AA), 8xAF, Default High settings, Mirrors on

Results:

Avg: 66.385

Min: 43

Max: 108

I must admit I'm completely new at ATI, I only have this card for a few days. I'm still quite lost in CCC. Normal antialiasing seems very weak compared to nVidia, there's lots of stuff that ordinary AA doesn't even try to smooth.

I'm toying with the new SuperSampling AA, on 2x it gives me much beter image quality than anything I have ever seen (it antialiases the whole screen, edges and textures), and the performance drop isn't big (3 FPS Average, Minimum stays the same). GPU gets warmer though, it stresses it at about 70% in DCS (compared to 60% at even 8x MultiSample AA). But the card is amazing, still only about 70 degrees C, and the fan remains very silent (below 35%).

But 4x SuperSampling AA results in more visible tearing, even if the FPS drop isn't that great, again only a couple of frames. I don't know why. Right now I'm a happy camper at 2x SuperSampling AA, 8x AF, Avg 64, Min 44, Max 107. And that's with mirrors on, which I normally switch off.

Very interesting.

Would you be able to (and confident) fiddle a bit with your memory clock? Running those sticks at 800MHz seems like a bit of a waste unless I've misunderstood something. Is that before or after allowing for the triple-channel?

Haven't really had the time to optimize the OC, I have to turn the CPU cooler (Scythe Mugen 2) and reapply the thermal paste since the temperatures shouldn't be that high.

800 MHz is shown in CPU-Z, it's actually 1600 MHz setting, nominal speed of these sticks. They should go up to 1900+ with relaxed timings, but I plan to lift them only by raising the base clock, right now I'm at 20 x 200, and multiplier for DRAM is 8x. And I'm sure processor can go higher, I'm giving him very little voltage (1.25 V CPU, 1,6 DRAM), and it's perfectly stable. Only a little hot with synthetic tests like Prime95 and Linx (above 75 degrees C).

rgr, makes sense then. Thanks. :)

I'm full of fail today: I launch the track and watch the chopper flop onto the ground after a few minutes and just sit there. I'm not giving it any input unless it's picking up something from the HOTAS. (if that makes any difference)

I'm also assuming the whole thing needs to be left at the default cockpit view as changing views would ruin the benchmark.

Help!

-Doc

EDIT: Hello?

Here are my results, from my not so stellar system (specs in sig).

Reducing AA to x4 had negligible affect on frame rates.

And yes, those flares are a real frame killer here too!

Preferred settings:

Textures HIGH

Scenes MEDIUM

Civ Traffic NO

Water LOW (EDITED IN .LUA)

Visib Range MEDIUM

Heat Blur ON

Shadows ACTIVE PLANAR

Resolution 1440x900

Aspect 16:10

Monitors 1 SCREEN

Res. of Cockpit Displays 256

Mirrors OFF

V Sync ON (nHancer)

Anisotropic Filtering 16x (nHancer)

Anti-Aliasing 8xQ Multisampling (nHancer)

Results:

Frames: 50468

Time: 1189628

Min FPS: 16

Max FPS: 61

Avg. FPS: 42.42

IBM Aptiva 500

Intel 400 MHz Processor

96 Mb Ram

Onboard Video

#1

Settings

1920x1200

Default High, Cockpit @ 1024, Mirrors On;

4xAA, 8xAF:

Results

Min: 47

Max: 97

Average: 71

;)

I'm full of fail today: I launch the track and watch the chopper flop onto the ground after a few minutes and just sit there. I'm not giving it any input unless it's picking up something from the HOTAS. (if that makes any difference)

 

I'm also assuming the whole thing needs to be left at the default cockpit view as changing views would ruin the benchmark.

 

Help!

 

-Doc

 

EDIT: Hello?

What was your version of DCS at the time? 1.0.1 is the way.

Hey EtherealN, I installed Windows 7 64bit not long ago and I wanted to re-run the same track again to check same settings and compare XO with Windows 7 but the track does not play correct (shows you going in circles)

.....but the track does not play correct (shows you going in circles)

Strange that - I benchmarked on Windows 7 as per my post and did not have any issues.

Not the first time this has happened either with tracks not playing back correctly/at all.......:music_whistling:

ah found out why... I had my helmet targeting circle in the middle of the screen which messed up the targeting and the track :) Anywho, I'm glad I found what messed it up

I've updated my post with benchmarks with XP vs Windows 7 64bit performance... looks like some 10% increase in performance with Windows 7 64bit on exact same hardware and game settings.

ah found out why... I had my helmet targeting circle in the middle of the screen which messed up the targeting and the track :) Anywho, I'm glad I found what messed it up

 

I've updated my post with benchmarks with XP vs Windows 7 64bit performance... looks like some 10% increase in performance with Windows 7 64bit on exact same hardware and game settings.

I thought Vista/7 gave a much larger improvement over XP due to multicore utilisation??

My post with benchmarks in this thread has exact same hardware and game settings only different OS... also take with grain of salt general people's claims how so much faster it runs etc etc... naturally when someone gets new PC or upgrade they want it to run lot faster to what they had previously otherwise they feel they've wasted money so people generally overrate how fast the system runs etc... but numbers don't lie

Yeah, the point is that Vista/7 can give a big boost compared to XP, if a lot of variables are right.

So far, the best tests I've seen were run by Maximus_G a long while back, testing XP, Vista and an RC of 7 against each other with full FPS graphs. Improvements were there, and quite significant, but there is a shortage of tests where identical hardware is tested with the OS as the only difference.

ok here we go. It seems I have a crap edition 5870 XXX card... have to underclock memory to 1275 (instead of factory 1300) ... so running I7 @ 4ghz... 5870 @ 900 1275

Windows 7 ultimate

min 0

max 201

avg 66, 594

AA 4 x

AF 8 x

RMA`ing the card though.

and again @ 1680 1050

Settings HIGH

1024 cockpits

Min, Max, Avg

22, 133, 76.484

i7 @ 4ghz - xfx 5870 @ 875 1250 ... poorly card ;)