Since the days of the early serial mainframes, parallel buses have dominated multiprocessing (MP) connections. But now, parallel MP buses will be edged out by very high-speed, pseudoserial switch fabrics. These will provide a more flexible, scalable, and reliable interconnect than standard, multidrop parallel buses. Memory-bus-to-memory-bus transactions will become packetized and will be put into specialized protocols to compete for networked bandwidth.
Switch fabrics like InfiniBand aren't only for linking system and networked servers. Expect to see them also at the processor chip level and at the board level on up, because switch fabrics aren't just the domain of system vendors. After all, it was the microprocessor chip vendors—companies like Intel and Motorola—that pushed packetized, pseudoserial bus connections. Intel launched Next Generation I/O (NGIO), the channelized, pseudoserial switch fabric that eventually mutated into InfiniBand. Furthermore, it was Motorola, with Mercury Computer Systems and others, that created RapidIO, a chip- and board-level switch fabric.
This shift to serialized, switch-fabric connectivity involves more than simply changing bus technology. For one thing, MP hardware systems will take on the characteristics of networked systems implementing protocol stacks of processing.
For another thing, connections will become more dynamic and scalable, enabling designers to easily add another compute or storage resource to their system. Plus, switch fabrics will help integrate computer and datacom/telecom systems. For the first time, both will use the same kind of serialized interconnect, differing only in the protocol stack that each implements. These emerging high-speed pseudoserial interconnect systems include:
- InfiniBand: a pseudoserial switch fabric for system-area networks (SANs). It supports both switch and router functions, and it can be de-ployed at the intrasystem, subsystem, and chassis levels. InfiniBand supports wide MP systems, but it may not be suitable for systems that require a real-time response. The specification is now complete, and silicon and software will be out in 2001.
- RapidIO: a pseudoserial switch fabric supporting processor-, board-, and chassis-level interconnects. MP vendors, like Mercury and Sky Computer, back RapidIO as the next-generation MP switch fabric for processors like Motorola's G4. This specification is under development.
- StarGen: a pseudoserial, high-speed switch fabric that supports board- and chassis-level interconnects. It provides a high-speed alternate bus that serves as a high-speed virtual PCI. StarGen enables MP systems with traditional PCI to move large amounts of data between nodes. Silicon and software will be available in the first quarter of next year.
- GigaBridge: a pseudoserial, high-speed bus architecture that supports board- and chassis-level interconnection at the PCI level. It provides a virtual PCI interconnect that links nodes with a PCI bus connection. GigaBridge employs a dual, counter-rotating ring to connect nodes. It adds more data bandwidth for traditional PCI-based servers and edge computers. Silicon and software will also be out in the first quarter of next year.
The essential characteristics of these new switch-fabric interconnects are scalabililty and flexibility. These switch fabrics are free from the limits of multidrop parallel buses. Unlike a parallel bus, a switch fabric doesn't have a fixed bandwidth or connection limits. It's made up of endpoints that connect through a fabric of interconnected node switches. These node switches are intelligent crossbars and the link between port-in message traffic and port-out message traffic. Additionally, each node switch, with its ports, has a built-in bandwidth limit of n/2 ports multiplied by the amount of bandwidth per port. Switch fabrics aren't new to MP systems. Both RACEway and SKYchannel delivered parallel switch fabrics (see "Early MP Switch Fabrics," p. 84).
But unlike a parallel bus, the fabric is extensible. If you need more bandwidth, just add more paths and switches. The fundamental limit on a switch fabric, though, is the aggregate bandwidth of its endpoints surrounding the fabric and the accumulated latency generated when moving through each switch node to the ad-dressed endpoint.
Recognizing the move toward a networked world, developers of the InfiniBand switch fabric chose to create an interconnection technology for networked systems (Fig. 1). This switch fabric goes beyond only supporting point-to-point dynamic connections between endpoints, or host CPUs and peripherals. It also supports both switches (transfers in a local subnet) and routers (transfers between different subnets). Therefore, InfiniBand can handle local, campus, and even wide-area connections between its endpoints. It's capable of very wide MP operations. Furthermore, InfiniBand drives up to 100 meters of cable. For larger WAN-like installations, however, communications links are needed between the InfiniBand segments.
But InfiniBand does more than only provide a network switch fabric with WAN-class connectivity. It ups processor and server I/O efficiencies too. InfiniBand adds channel capability to standard server processors.