Inside AnandTech 2013: Power Consumption

Two of the previous three posts I've made about our upgraded server infrastructure have focused on performance. In the second post I talked about the performance (and reliability) benefits of going with our all-SSD architecture, while in the third post I talked about the increase in CPU performance between our old and new infrastructures. Today however, it's time to focus on power consumption.

Our old server infrastructure came from a time where power consumption mattered, but it hadn't yet been prioritized. This was before Nehalem's 2:1 rule (2% perf increase for every 1% power increase), and it was before power gating. Once again I turned to our old HP DL585 server with four AMD Opteron 880s (8-cores total) as an example of just how things have changed.

As a recap, we moved from the DL585 (and over 20 other 1U, 2U and 4U machines with similar, or slightly newer class processors) to an array of 6 Intel SR2625s (dual-socket 6-core Westmere based platforms), with another 6 to be deployed this year. All of our previous servers used hard drives, while all of our new servers use SSDs. The combination resulted in more than a doubling of peak CPU performance, and an increase in IO performance of anywhere between a near tripling to over an order of magnitude.

Everything got better, but the impressive part is that power consumption went down dramatically:

With both machines plugged in to a power outlet but both completely off, the new server already draws considerably less power. The difference at idle however is far more impressive. Without power gating and without a clear focus on minimizing power consumption, our old DL585 pulled over 500W when completely idle. It shocked me at first, but remembering back to how things used to be back then it stopped being so surprising. There was a time when even our single socket CPU testbeds would pull over 200W at idle.

Under heavy integer (7-zip) and FP (Cinebench) workloads, the difference is still staggering. You could run 2.5 of the new servers in the same power envelope as a single one of the old machines.

The power consumption under heavy IO needs a bit of explaining. We were still on an all 3.5-inch HDD architecture back then, so we had to rely on a combination of internal drives as well as an external Promise Vtrak J310s chassis to give us enough spindles to deliver the performance we needed. The 693.1W I report above includes the power consumption of the vTrak chassis (roughly 150W). In reality, all of the other tests here (idle, 7-zip, Cinebench) should include the vTrak's power consumption as well since the combination of the two were necessary to service the needs of the Forums alone. With the new infrastructure everything can be handled by this one tiny 2U box. So whereas under a heavy IO load our old setup would pull nearly 700W, the new server only needs 170W.

Datacenter power pricing varies depending on the size of the customer and the location of the datacenter, but if you were to assume roughly $0.10 per kWh you'd be talking about $459 per year (assuming 100% idle workload) for our old server compared to $92.50 per year for the new one. That's a considerable savings per year, just for a single box - and assuming the best case scenario (also not including the J310s external chassis). For workloads that don't necessarily demand huge increases in performance, modernizing your infrastructure can come with significant power and space savings (not to mention a positive impact to reliability).

Keep in mind that we're only looking at a single machine here. While the DL585 was probably the worst example from our old setup, there over a dozen other offenders in our racks (e.g. dual socket Pentium 4 based Xeons). It's no wonder that power consumption in datacenters became a big issue very quickly.

Our old infrastructure at our old datacenter was actually at the point where we were power limited. Although we only used a rack and a half of space we had to borrow power from adjacent racks because our requirements were so high. The new setup not only allows us better performance, but it gives us headroom on the power consumption side as well.

As I mentioned in my first post, we went down this path back in 2010 - there have been further power (and performance) enhancements since then. A move to 22nm based silicon could definitely help further improve things. For some workloads, this is where the impact of microservers can really be felt. While I don't see us moving to a microserver environment for our big database servers, it's entirely possible that the smaller, front-end application servers could see a power benefit. The right microprocessor architectures aren't available yet, but as Intel moves to its new 22nm Atom silicon and as ARM moves to 20nm Cortex A57/A53 things could be different.