Virtualization is receiving lots of attention these days. Behind the buzz are some simple, time-tested concepts. But the movement of this technology from the mainframe to the mainstream has brought it into the limelight.
At its heart, virtualization is about making something “look” like something else. Typically, this means making an operating system “think” it’s running alone on a computer, when in fact that computer is shared by several operating systems—each referred to here as a system image (SI).
Since mainstream computers started incorporating memory-management units, this type of virtualization has been possible, albeit not terribly popular due to the extreme performance hit required to emulate every device in the computer. Recent hardware and software technology has boosted emulation speed, but plenty of room for improvement remains.
I/O virtualization (IOV) is simply the ability to make one device look like multiple devices, each assignable to a unique SI. Moving the virtualization to the device level can provide dramatic performance increases by freeing the system processor(s) from the cumbersome task of emulating those devices.
The PCI-SIG has defined a mechanism for providing these virtual device interfaces on the PCI Express (PCIe) bus. Now that these efforts are complete, some needed standardization is finally available in mainstream IOV design, enabling multi-vendor silicon solutions to work on multiple platforms under multiple operating systems.
Problem solved, right? Take a chip’s current PCI Express front-end logic, slap on this I/O virtualization stuff, and “poof!” It’s virtualizing like mad. Well, yes and no. Yes, it creates a chip assignable to multiple SIs at once. However, chances are that virtualizing the back end of the device is actually the harder part of the problem.
Also, consider a storage controller— clearly, one would want to partition the connected storage so System Image X (perhaps an online banking system) has space allocated only to itself and isn’t accessible by System Image Y (e.g., the Web server for www.hackers-are-we.org).
PCI EXPRESS I/O VIRTUALIZATION Let’s take a brief look at the system view of IOV. The term “system image” refers to a real or virtual system of CPU(s), memory, operating systems, I/O, etc. Multiple SIs may run on one or more sets of actual hardware.
One example today might be a hypervisor like VMWare running Windows XP and Linux simultaneously on a single-CPU desktop PC. In that case, two SIs exist, each sharing a single CPU, memory, disk drive, etc. Another example would be a blade server running Windows XP on one blade and Linux on another blade. There, each SI isn’t sharing any of the CPU blade’s hardware, though it could potentially share hardware on an I/O blade.
Regardless of the physical assignments, each SI needs to “see” its own PCI hierarchy. Even if no end devices are actually shared (e.g., two Fibre Channel controllers on the I/O blade, one assigned to the Linux blade and one assigned to the XP blade), some control over the hierarchy’s visibility is required. If end devices are shared, each SI must be restricted to seeing only its “portion” of shared end devices.
The device needs to make its one physical set of hardware appear to be multiple virtual devices, which appear completely independent to outside observers. Those devices may:
• occupy different PCI memory ranges • have different settings for various PCI configuration registers • potentially each be a particular PCI multifunction device
Furthermore, the device needs to keep cross-“device” traffic isolated internally so no data spillover occurs between virtual devices.
As seen in the examples above, a clear distinction can be drawn between systems having a single point of attachment to the PCI hierarchy and those with multiple points of attachment. The traditional single-CPU desktop computer and even the traditional n-way multi-CPU server previously had a “single” logical point of attachment to the PCI hierarchy (Fig. 1).
By contrast, blade systems enable a new hierarchy view where some upper-level enhanced PCI Express switch could allow multiple root complexes to attach to the total PCI hierarchy (Fig. 2). Here, some new mechanisms are clearly required to enable each root complex to access only its assigned portion of that hierarchy.
Given the large separation between these two types of systems, both from a complexity and market segmentation perspective, the PCI-SIG chose to break IOV up into two separate specifications. Since each root complex (Fig. 2, again) could also be utilizing single-root IOV, the two specifications will necessarily be interdependent. Thus, the so-called “concentric circles” model was adopted, whereby the single-root specification builds on the PCI Express base specification, and the multi-root specification builds on both the single-root specification and PCI Express base specification.
Reprints
