Hardware Virtualization: the Nuts and Bolts

Conclusion

When we first heard about Intel's VT-x and AMD's SVM technology we expected to see performance improvements over the software based solutions such as Binary Translation and Paravirtualization. Both AMD and Intel gave the impression that they were about to "enhance" and "accelerate" the current purely software based solutions.

What AMD and Intel did was extend x86 to make "Classic Virtualization" possible, very similar to the old IBM mainframe virtualization. In other words, hardware virtualization support does not really "enhance" Binary Translation or Paravirtualization; it is a completely different approach. First generation hardware virtualization was even a step back from a performance view, but one that enabled many steps forward. The first generation of virtualization has been improved, and is now adopted by VMware to support 64-bit guest OSes and by Xen to run unmodified OSes (such as Windows). So right now, hardware virtualization still has a long way to go while software virtualization is mature and represents the current standard.

Second generation virtualization (VT-x+EPT and AMD-V+NPT) is more promising, but while it can improve performance significantly it is not guaranteed that it will improve performance across all applications due to the heavy TLB miss cost. On the flip side of the coin: software virtualization is very mature, but there is very little headroom left to improve. The smartest way is to use a hybrid approach, and that is exactly what VMware, Xen, and Microsoft have been doing.

VMware ESX is the best example of this. ESX uses paravirtualized drivers for the most critical I/O components (but not for the CPU), uses emulation for the less important I/O, Binary Translation to avoid the high "trap and emulate" performance penalty, and hardware virtualization for 64-bit guests. In this way, virtualized applications perform quite well, in some cases almost as if there is no extra layer (the VMM).

That doesn't mean that there are no performance issues at all. The huge number of people that populated the numerous VMWorld 2008 sessions about performance and our own (early) benchmark results tell us that the real world performance of virtual servers is still a very interesting challenge despite the multi-core powerhouses they are running on. As long as you are running CPU intensive applications, there is no problem at all - they are running directly after all. However, consider applications with one or more of the following characteristics:

• Have a high frequency of system calls or interrupts
  
• Access the memory intensively (DMA)
  
• Perform a lot accesses to I/O devices
  
• Require SMP to perform well

These applications will need more attention to perform well on a virtualized server. Now, it's time for more in-depth benchmarking. Stay tuned….

Bibliography

^[1] Robin/Irvine, 9th USENIX Security Symposium Paper 2000: "Analysis of Pentium's Ability to Support a Secure VMM"

- http://www.usenix.org/events/sec00/full_papers/robin/robin_html/index.html

^[2] Scott Devine, Co-Founder & Principal Engineer VMware, Inc., "Introduction to Virtual Machines Introduction to Virtual Machines"

^[3] Ole Agesen, VMware, "Performance aspects of x86 virtualization", VMWorld 2008

^[4] Keith Adams, Ole Agesen, VMware "A Comparison of Software and Hardware Techniques for x86 Virtualization"

^[5] Daniel P. Bovet, Marco Cesati, "Understanding the Linux Kernel 3rd Edition" , O'Reilly

November 2005, ISBN: 0-596-00565-2, section 1.6

^[6] Paul Barham , Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer , Ian Pratt, Andrew Wareld, "Xen and the Art of Virtualization", University of Cambridge Computer Laboratory