|
Synthetic VS Real World Benchmarks
Why the Debate?
You may have noticed that many hardware and software review websites make sure to note when they use real or synthetic benchmarks in a particular review. Some sites concentrate solely on synthetic benchmarks while others concentrate on the so called "real world" benchmarks. The important thing is not to know which one is right or wrong but to understand what the difference is and what you can personally do with the data that you get from these benchmarks you find online. So lets first start with the difference between real world and synthetic in terms of benchmarking.
An often overlooked fact is that 99% of all the people that will read a review are unable to attain the same results on their personal machines as in the review. This is because almost all reviews are done on a machine with a fresh install of the operating system with as little running in the background as possible. In many cases reviewers are able to artificially inflate scores by disabling hardware devices such as audio, USB, and IDE which free up the CPU and system to run faster. While this practice does not reflect an accurate picture of system performance it is a perfect example of why "real world" benchmarks are so important. In the real world a particular system may be used for several years and have a lot of different software loaded into it. These utilities and applications are largely responsible for system slowdown and are why your particular system may score so much lower than what you see online.
Synthetic Benchmarks
Traditionally the benchmarks you find in 90% of the reviews online are what is known as synthetic in nature. This means that the benchmarks itself does not actually perform a function outside of producing numbers and data used to provide a comparison. An example would be Sisoft Sandra and 3DMark06 and 3DMark Vantage. These programs exist so you can compare your machine against other sets of machines already benchmarked. The number provided by 3DMark06 for example gives relative performance and in and of itself does not give you a useful idea of the performance of the system or component. For example lets say we used benchmark A on a test system and received a score of 20,000 points. What does that score tell you? It tells you nothing unless you have another system, hopefully several, to compare it against. It is only when this data is compared against other systems the hardware for which is known does the data begin to provide a sense of positioning or performance.
Synthetic benchmarks certainly have their usefulness since the data and perofrmance is capable of being replicated on several different machines. It is also useful since in the world of reviews the reader very rarely has a system identical to that in the review. In such cases the synthetic benchmark allows the reader to test their system against the numbers provided in the review to gain an idea once again of relative performance.
Examples of Synthetic benchmarks are:
3DMark Vantage: 3D Benchmark targeted towards the video/graphics card.
Sisoft Sandra: A set of benchmarks packaged together that determine component performance and deliver an overall system score.
CineBench R10: Runs several tests on your computer to measure the performance of the main processor and the graphics card under real world circumstances. The benchmark application makes use of up to 16 CPUs or CPU cores and is available for Windows (32-bit and 64-Bit) and Macintosh (PPC and Intel-based).
SuperPI: A computer program that calculates pi to a specified number of digits after the decimal point - up to a maximum of 32 million.
LightsMark: Benchmark/demo with REALTIME GLOBAL ILLUMINATION and PENUMBRA SHADOWS.- PCMark: PC Performance Analysis benchmarking software which runs a suite of tests designed to report an overall system score. Runs exclusively on Windows Vista.
HD Tune: Hard drive benchmark designed to test, write benchmark, secure erasing, AAM setting, folder usage view, disk monitor, command line parameters and file benchmark.
HD Tach: HD Tach is a low level hardware benchmark for random access read/write storage devices such as hard drives, removable drives (ZIP/JAZZ), flash devices, and RAID arrays.
I/O Meter: I/O subsystem measurement and characterization tool for single and clustered systems.
Crystalmark: A set of benchmarks packaged together that determine component performance and deliver an overall system score.- Whetstone: CPU based benchmark centered around floating-point arithmetic performance.
- Dhrystone: CPU based benchmark centered around integer arithmetic performance.
We have provided links to each of the software listed above so that you can compare your system results to those found online.
Real World Benchmarks
On the opposite end of the spectrun are real world benchmarks. These are traditionally programs that began life offering functionality not aimed at providing scores or numbers. Examples of which are audio encoding programs, video and 3D rendering, office applications, and video games. These programs when used as benchmarks are known as real world benchmarks because they are applications that the traditional users will actually run in every day life. Few people outside of game, benchmark developers, and hardware manufacturers, create a system solely to be a benchmark platform and deliver scores on. The average consumer buys or builds a machine to run office applications, play games, create video and audio, and to access information both online and locally. So for the average consumer the real world benchmarks carry much more meaning but are often overlooked.
Real World benchmarks are much more difficult to interpret for the average consumer as they are much more subjective. Variables are introduced into running these benchmarks which can wildly affect the results. Most websites do not provide enough information when reporting real world benchmarks to enable someone to accurately read the results and place them into perspective. Websites like HardOCP have been big proponents for real world benchmarking and trying to educate people on the need for such results.
Examples of "Real World" Benchmarks:
| CineBench R10: | Runs several tests on your computer to measure the performance of the main processor and the graphics card under real world circumstances. The benchmark application makes use of up to 16 CPUs or CPU cores and is available for Windows (32-bit and 64-Bit) and Macintosh (PPC and Intel-based). | |
 |
||
| Encoding LAME | LAME is considered the best MP3 encoder at mid-high bitrates and at VBR and is probably the only MP3 encoder still being actively developed. |  |
| Adobe Photoshop | Photoshop CS3 includes tools for editing 3D and motion-based content and performing image analysis. |  |
| DIVX Encoding | DivX technology compresses video to a fraction of its original size for efficient storage on your hard drive and easy sharing online. |  |
| Game Benchmarks |
|
 |
So what is more important, Real or Synthetic benchmarks?
The answer is not so clearly defined, however, in terms of usefulness "Real World" benchmarks can give a much more accurate picture of system performance. When interpreting the results you find on our site as well as on others it is critical that you take into consideration all of the system components used in the "Test" system as well as if the software used is synthetic or real world in nature. Good hardware and software review sites will give full disclosure of all the system specifications as well as software used in the review. This should include the following:
- Operating System and service packs used
- Drivers Used
- Test System Configuration
- CPU
- Hard Drive
- Motherboard
- Ram
- Video Card
- Settings Used Per Benchmark
- An interpretation of the results
Now that you know the difference between these types of benchmarks and what to look for we hope that you will have an easier time reading reviews online.