Test methodology

Background

There are four tests in PPBM6 just as in the previous version. The timeline has been modified in comparison to PPBM5 to better reflect system load in all aspects and to improve statistical accuracy.

  1. Render the Timeline to create Preview files (Pressing Enter).
    This test may have to be done twice, once with Hardware MPE acceleration and once with Software MPE only.
  2. Export the Timeline with Abode Media Encoder to a MPEG2 DVD file.
  3. Export the Timeline with Adobe Media Encoder to a H.264 file.
  4. Export the Timeline with Adobe Media Encoder to a Microsoft DV AVI file.

These tests were selected because they fully test the performance of the disk subsystem and the CPU, memory and GPU system.

  1. The Rendering of the H.264 Test Timeline is mainly oriented to measure Mercury Playback Engine benefits. If hardware MPE is not available, it is used to measure overall performance.
  2. The Export to MPEG2 DVD from the MPEG2 DVD Test timeline is very CPU/memory intensive.
  3. The Export to H.264-BR from the H.264 Test timeline is very CPU/memory intensive.
  4. The Export to DV-AVI from the Disk Test timeline is a very disk intensive test.

Methodology

The Total Time score is the sum of all individual tests. Not all systems are built equally, some may have a super fast CPU, others may compensate a slower CPU with a faster video card or more memory and still others may have multiple disks or raid configurations. To get a feel for real-life performance, a performance index has been added, which will show you your system's performance relative to the top-ranking machine in daily life. Using weights for the four different tests, normalized scores for each machine are calculated and indexed to the top-ranking system to show the difference in performance.

DISK I/O test:

The overriding factor is disk speed here.  The test uses many small reads and a large sequential write (nearly 19 GB). Number of cores makes no real difference (it is not well multithreaded), but clock speed does.

MPEG2 DVD test:

The two overriding factors here are amount of memory and number of cores. More is better here. Additionally the location and speed of the pagefile can be important especially if you have a small amount of RAM.

H.264 test:

Here the speed of CPU/RAM communication is king. Number of cores, clock speed and the amount of CPU cache are very important. Dual processor systems are hampered by the 2 chip communication.

CPU / GPU Test Result:

This is almost solely based on the video card and whether hardware or software MPE is used.

MPE Gain:

This shows how much faster hardware MPE rendering is than software only rendering. The minimum score is of course 1, since if there is no hardware MPE available, there is no performance gain.

CAVEAT:

A high score does NOT mean better. If the memory or CPU is relatively slow, the MPE Gain is bigger and vice versa, if the CPU and memory are very fast, the MPE Gain is lower. So interpret with care.

Second CAVEAT:

On export hardware MPE uses maximum quality, which results in much better scaling than software only, so the MPE Gain is not only speed, but also quality.

Total Time:

The Total Time is the sum total of the individual test scores, where only the lowest of the MPE scores is counted, as shown in the CPU / GPU Result column.

Relative Performance Index:

To avoid a heavy impact of any single test on the total time, the four test results each have a performance weight attached and each result is normalized against the top ranking machine. In this way each test has the same impact on the RPI, despite the sometimes large differences in measured speed. Simply said, the RPI shows how much slower a system is than the top-most system we have encountered.

On a properly balanced system, both the Total Time and the Relative Performance Index will show around the same distance from the top machine. So a Total Time slower by 20% and a RPI of 1.20 means well balanced.

Our Hardware Design Conclusions

  • Absolutely most important is the CPU, right now that is the Intel i7 processor or the dual processor Xeon 56xx series. Specifically for MPEG encoding the amount of memory is the most important facor to influence performance. As you can see from the performance data the speed of the processor is also very significant. AMD processors lack the full SSE set of instructions and are not as effective as the Intel processors.

CPU-Clock  

  • A dual processor setup profits from the extra cores during MPEG encoding, but is hampered during H.264 encoding due to latencies between the 2 chips, a single i7 profits from more cache.
  • With CS6 the practical minimum memory is 8 or 12 GB of RAM, 16 or 24 GB is suggested for most users. For heavy multitasking or dynamic linking, 32 or 48 GB is optimal.  Here is a Resource Monitor plot of memory usage while doing a simple PPBM5 MPEG2-DVD encoding in Premiere.  This was a capture at a peak of memory usage.  Notice that there are no Hard Memory Faults (requiring  writing out and later reading the Paging file) but Premiere by itself is using more than 6 GB (this system had 12 GB).

  • SSD's are nice for laptops, but in editing desktops they do not show any benefit in performance.

  • The Adobe minimum basic disk system is an absolute minimum of two 7200 rpm disk drives.  Our personal preference is for a 10,000 rpm drive for the Operating System & Applications disk and a RAID array for the project files. Specific functions like a separate dedicated drive for writing Output files or Preview files are of less value as it may just slow things down compared to a high performance RAID, the only way to tell is run the PPBM6 benchmark.

  • A CUDA/MPE card makes a huge difference in performance and improves quality of the output over software MPE.