Real world benchmarking is an misnomer!

[Kirby]

I was reading Bit-tech.net today and found a review of the new successor to Thermalright’s TRUE 120, Thermalright Venomous-X. They gave a bad review for it, not because it performed bad, but rather because it didn’t perform better than its competition, while costing more. Fair enough. However, littered throughout the comments are people talking about the testing setup, and how it does not accurately reflect "real world" conditions. Often bringing up HardOCP's methodologies. Some suggestions are good, but overall it just seems people want the reviewers to test their world. That IS NOT how you benchmark.

If your real world and benchmark numbers do not correlate, then your benchmarking methods are wrong. That’s correlate, not exactly the same. Benchmarks will rarely, if ever, give you numbers that match exactly what you get at home.

A benchmark should tell me statistics, indexes of how it will perform under certain conditions. For example a heatsink: how it performs under high/low wattage, high/low cfm, and maybe high/low ambient. All other factors should remain exactly the same. The case, the fan, and the high/low number should all be the same. Unless I’m missing something, I don’t believe you even need a CPU. A heating element with a CPU top, and a thermal probe in it would do just as well. For all intents and purposes, your heatsink is just moving heat away from your CPU. It doesn't care what GHz it's running at. Nor does it care what brand it is. As far as the CPU goes the only thing that it cares is how much heat the thing is throwing out, and the surface it's contacting.

As for the “real world” argument from HardOCP, I believe they made the change when their benchmarks gave them a conclusion for a videocard that should have been smooth gameplay. However from their end user point of view, it wasn’t. I understand the reason they made the change, they didn’t want to give a good review for something that isn’t. But I believe they went the wrong route in fixing this issue. Now, the video benchmarks are largely subjective. Their world is not the same as mine, and what they believe to be good enough may not be for me. Even if I were to takeout the subjective part, it’s become extremely difficult to discern from their data which is actually better and if so, is it worth the price difference. 4AA @ 40fps vs 8AA @ 30fps, while also one is 1920 but the other is 1680? That is waaaaay to many variables.

Instead of changing the whole benchmark, they should have looked for the reason why and improve upon the original benchmark. Why was the gameplay jittery even though the FPS would indicate otherwise? Could there be something else that we need to test for that we’ve never noticed in the past?

Can you imagine if SSD reviewers never looked at small non-sequential transfer rates when they noticed stuttering? What if Anandtech said “I’m going to IM a bunch and see which feels smoother”? That would be real world, but would it tell you how much better an Intel is compared to OCZ?

Maybe that wasn't possible? Well I still question the "real world" testing methods. I don't go to a review to tell me what setting I can play a certain game on a certain videocard at. I go to tell me which hardware is better, and by how much. I don't care if it doesn't reflect the real world as long as it reflects how it would compare to other hardware. That is how I purchase.
_________________
"Teamwork is a lot of people doing what I say."