Testing the latest x86 rack servers and low power server CPUs

Idle Power in ESX: Challenging

Measuring idle power in ESX is easier said than done. ESX does not use SpeedStep or PowerNow! by default (similar to Windows and unlike Linux). You have to set the power.cpu policy to "dynamic" instead of the static default. That means that many virtualized data centers are consuming more than they need to as they are not using SpeedStep or PowerNow! Getting PowerNow! to work was pretty easy. SpeedStep was a different matter: only our Intel Willowbrook based server (Chenbro chassis) could enable it. We of course first enabled SpeedStep in the BIOS (EIST = enabled):

However, even after restarting the machine several times and setting power.cpupolicy to dynamic we ended up with this on both the ASUS and Supermicro machines:

Notice that the Power Management technology is not available. The Intel Willowbrook based server did not have this problem:

* = SpeedStep was not supported.

Conventional wisdom from a few years back might make you expect that the lower clocked parts (like the Xeon E5504 80W) consume less power than the highest clocked parts (like Xeon X5570 at 2.93GHz), but that is far from the truth. In fact, the "basic Xeons" are the leakiest parts - the parts that Intel almost threw in the trash bin.

The idle numbers show that Intel's power gating techniques save quite a bit of energy. Notice how adding a second L5520 CPU and three DIMMs of DDR3-1066 to our Chenbro server only adds 9W. Add an Opteron EE to our AMD server and you add 22W. According to AMD's own numbers, the lowest idle power measured for the quad-core Opterons is about 13W, probably using a Windows Server version that is patched with the best power saving features available. A quick comparison shows that VMware's ESX is not that aggressive when it comes to saving power, so the idle CPU power is quite a bit higher. The result is that the best AMD platform consumes about 16W more than the best Intel platform when running idle.

Granted, a better AMD platform can lower that number, but it is clear that when it comes to idle power, the platform is not the only one to blame here. Intel's power gating is the decisive advantage here: it can turn the inactive cores completely off. Another indication is that the dual Opteron 2435 consumes about 156W when we turn off dynamic power management, which is higher than the Xeon X5570 (150W). On the flip side of the coin, AMD's "older" platform makes enabling PowerNow! easier in ESX 4.0.