Fortunately, it takes absolutely no changes in software to establish and utilize this system. The existing BIOS firmware and operating system are used without modification. In this example, there are now five PCI bus segments, which could each have five to eight devices in one monolithic system—all communicating through one address space.
This mode enables lots of real-world applications. One is supporting a remote chassis of PCI expansion slots for an existing server or workstation that's currently limited by the number of slots in its chassis. Because StarFabric nodes can be connected with five meters or more of CAT 5 cable, a compute chassis can be in a separate enclosure, remote from its noisy disk drives—possibly in an adjacent room.
Advanced Applications: As designers look to build larger and more diverse systems, like communication platforms with StarFabric, the pure-legacy, address-routing mode becomes a limitation. The hierarchical system map requires all traffic to flow through the root of the tree (Fig. 2, again). Also, legacy PCI address-routing methods don't take advantage of all present interconnects between StarFabric nodes. For larger systems, traffic must first flow up to the root/host node, then to its destination.
Many developers will want to benefit from features that eradicate these limitations, and others that provide QoS, high-availability, and address/device isolation at each node for distributed-computing applications. For these reasons, it may be desirable to move to path routing and use the gateway function in the PCI-to-StarFabric bridge.
Conceptualize the gateway function within the bridge as an embedded PCI-to-PCI bridge. With it, devices, including processors, can be "hidden" behind the bridge. Each processing element can have its own address space and control its own devices. Communication across the fabric is done by connections set up at system initialization. A connection specifies the route through the fabric, from a single device to every other device with which it needs to communicate.
In fact, multiple paths from every source to each destination can be stored. A primary path is normally used, but in case of a problem somewhere on that path, a secondary path specification can be employed. This high-availability feature can be set to either fail-over from the primary to secondary route automatically in silicon, or to fail-over with software intervention. In either case, event notification is supplied to an event handler whenever a path status changes within the fabric.
During path routing, the system boot-up process picks up where address routing left off. Once the initial enumeration "storm" has established the unique fabric identification for each fabric node, the BIOS and operating system create the PCI hierarchy view of the system—and the boot process continues. At this point, a software object called the Fabric Programmers Library (FPL) is invoked, and the path-routed connections desired by the system designers are established. The FPL offers a wide range of capabilities.
Path routing lets designers build highly capable distributed-computing systems. It supports sophisticated schemes for handling system events and interrupts. The system can be devised to control devices and locate drivers anywhere in the system. Any processor can be set up to handle interrupts, with various processors working with interrupts from different devices.
Only a local memory read is necessary to identify an interrupt source. In this environment, the system knows exactly where these interrupts came from, and sometimes, how to deal with them. In a large-scale system, this interrupt-source identification alone provides two to three orders of magnitude lower latency for interrupt handling than a conventional system.
Moreover, developers can build priority queues for I/O interrupts—in contrast to the simple nonprioritized interrupt vectors of a conventional system. This ability to map and route interrupt priorities permits developers to implement interrupts in new ways.
Also, in-band interrupt schemes are far more efficient than PCI's out-of-band approach. For example, if an SCSI device sends an interrupt to a processor in a PCI system, the data might not have completely cleared the bus by the time the processor fields the interrupt. But interrupts can travel in-band via data in StarFabric, so data associated with the interrupt can be pushed ahead of it into memory, where the processor uses it immediately. Depending on the system map, the processor may not have to read devices at all, but just perform a write that clears the interrupt.
The path-routing operating mode permits the system to attain real QoS. Seven classes of service are supported by the StarFabric protocol: asynchronous, high-priority asynchronous, isochronous, high-priority isochronous, multicast, special, and provisioning.
A credit-based flow-control algorithm moves traffic through the system only when sufficient buffer space exists at the next node. Frames never need retransmission unless an error occurs. Congestion within one class, say asynchronous, won't block other more important classes, like isochronous. The programming interface grants system designers the flexibility to reallocate the credits of every class based on the specific application.
Aside from pure-legacy mode or path-routed systems, the technology handles hybrid systems of address and path routing. This way, designers can maintain the advantages of legacy-code support for many system functions and add the benefits of path routing where appropriate.
In summary, switch-fabric technology gives system designers a wide range of options and flexibility. They can trade the ease of design and reuse of existing software of address routing for the robust feature enhancements of path routing. Many will take an evolutionary approach where address routing and path routing coexist, with the percentage of each varying over time.