Testing the latest x86 rack servers and low power server CPUs

Power: Low Power CPUs Compared

First we'll start with the raw power consumption:

By dividing the performance scores on the previous page by the measured power consumption at full load, we get the performance per watt. We multiplied by 100 to make the results easier to read, so a score of 100 means that for every vApus Mark I performance point the system consumes one watt.

That the X5570 2.93GHz wins the performance/watt race came as a surprising… at first. However, remember that the "leakier" Nehalems go to the desktop as 130W TDP parts, while the server CPUs are the best parts (95W TDP). The top server Xeons are already binned for power consumption. In a way, the 2.93GHz Xeon is already a "lower power" CPU.

The "75W ACP Shanghai" based Opterons disappoint: the difference between them and the 95W TDP Xeon "Nehalem" is small. This is partly a result of the fact that the NVIDIA chipset based motherboards waste quite a bit more power than the Intel platform. Notice that a single Opteron 2435 consumes more than a Xeon X5570; once you add three DIMMs and a second CPU, the tables have turned. So adding a Xeon X5570 adds about 58W (248W - 175W - three DIMMs of 5W), while adding an Opteron 2435 2.6GHz adds about 47W (243 - 181 - three DIMMs of 5W). While the Opteron 2435 consumes less power than the X5570, an AMD based server still consumes slightly more than the Intel based server with one socket filled up. This clearly indicates that our AMD server platform is not as efficient as the Intel Server. This is something that AMD's upcoming high-efficiency Fiorano "Kroner" platform should address.

The Xeon X5570 2.93GHz only consumes a few watts more than the much slower quad-core Opterons. We have shown that in native situations, the Xeon X5570 is about 37% to 85% faster than a 2.7GHz quad-core Opteron or about 30% to 78% faster than a 2.9GHz Opteron. In our virtualization scenario, the X5570 is about 30% faster. The business case for the high-clocked quad-core Opteron looks very bleak. The six-core Opteron at 2.6GHz is stronger: 15% higher performance than the 2.9GHz quad-core and it consumes just as much. The Opteron EE performs well, although it fails to defeat the L5520 when it comes to "pure" performance/watt.

Power at full load is only part of the story of course. We are working on a throttled vApus workload, but a good look at the numbers from SPECpower_ssj2008 tells us that the power consumption at 60-90% is not radically different from 100%. That is not really surprising: 100% CPU load does not mean that the CPU is using twice the amount of decoding and executing resources than at 50%. Instead, 100% load means that the (ESX) scheduler never has to run an idle process. That happens 50% of the time when you are running at 50% load. Basically, you get the same resources stressed 50% or 100% of the time. You may expect that the power curve starting from the idle power value to the 100% load power value is more or less linear… if the clock speed and voltage is constant of course. By default, ESX does not change the clock speed (using SpeedStep or PowerNow!) as long as your load does not go under 60%. Whether we test at 60%, 80%, or 100%, the power consumption landscape should stay the same.

Some applications will always run at a relatively high CPU load, for example a web server that is read in Asia, Europe, and America. However, many applications are only used between 9AM and 6PM local time, so a large part of the time the machine might be running close to idle. Before we declare anybody the performance/watt king, we need to take the idle power numbers into account.