[Design Application]
PCI Scales New Heights With Switch-Fabric Interconnects
Choose a transparent-bridge mode for legacy applications, or a path-routing gateway function for advanced systems.
PCI and CompactPCI (cPCI) technology and systems are pervasive across a wide range of applications. But as these applications evolve, the limitations of both technologies are surfacing. At the same time, the investment in them can't be written off. Consequently, designers who push PCI and cPCI have hit limitations like the lack of scalability, Quality of Service (QoS), and high-availability issues.
Luckily, new switched-interconnect technologies take PCI to another level. One example is StarFabric, a scalable, universal switch fabric that addresses the unique requirements of communication engineers designing for next-generation data, voice, and video networks.
The need to scale systems is driven by telecommunication carriers' requirement to control expenses. Besides system-acquisition prices, rack-space cost and the cost to manage systems are gaining importance to carriers. All types of carriers are placing a premium on rack space, forcing system designers to increase port densities in their systems.
Similarly, as systems grow from one chassis to multichassis, the total package must look like a single unit to the management system. Current PCI-bridged solutions create both complex hierarchies and a significant increase in latency, which limits the ability to scale sufficiently. But a switched-interconnect solution eliminates this problem.
Evidently, the public switched-telephone network (PSTN), based on time-division multiplexing (TDM), won't go away anytime soon. Therefore, all kinds of systems must sustain both TDM and packet traffic. Today's solutions require separate TDM and packet buses. As noted, each bus needs to meet the necessary scaling.
But scaling is becoming a challenge. One approach is to combine the buses into a switched-interconnect technology, such as StarFabric, then send native TDM traffic and packet traffic over one switch fabric—with the TDM voice traffic having a higher priority. Plus, the 8-kHz timing reference can go through the fabric to retain synchronization.
In the telecom world, five-nines availability has always been the rule. But in the packet world, five-nines was solely a powerpoint concept. As the packet and TDM worlds "converge," though, the five-nines goal is becoming mandatory. Frequently, five-nines are table stakes with the real availability requirement of six-nines or more. If systems with separate TDM and packet buses ever meet the availability requirement, they will need redundant buses for each. This would take four buses within the system.
The cost and complexity involved is substantial. A more elegant solution is a single fully redundant switched-interconnect topology that provides the inherent QoS treatment of the TDM and packet traffic.
To meet these challenges, StarFabric technology is being designed into a plethora of communication and distributed-computing applications, ranging from voice-over-packet media gateways and optical edge routers to video servers and medical imaging platforms. All require the low cost and high scalability of a switched-interconnect technology. The built-in support for QoS—native TDM voice traffic, high availability, and loosely coupled distributed computing with efficient memory-to-memory communication features—makes it ideal for many compute and data-intensive applications.
There are two initial silicon components using StarFabric technology. One is a high-throughput switch (SG1010) that supplies 30-Gbit/s switching capacity with six ports. The other is a PCI-Fabric bridge (SG 2010) device that interfaces 64- or 32-bit PCI buses (operating at 66 or 33 MHz) to StarGen's switch fabric.
The fundamental physical-layer interconnect for the initial components is a 622-Mbit/s differential pair with a 400-mV swing. Each port consists of four of these pairs in each direction—yielding an aggregate bandwidth of 5 Gbits/s. This accommodates the bandwidth required by next-generation communications equipment. For the initial components, two of these ports can be bundled to provide a 10-Gbit/s "fat pipe" between endpoints.
Depending on the implementation, a switched-interconnect system can comprise any number of components. This gives designers the freedom to build a variety of systems, from very small-scale systems to very large-scale systems with hundreds of endpoints.
The PCI-to-StarFabric bridge is a multifunction device. It incorporates both a familiar transparent-bridge function and a fabric native-gateway function (Fig. 1). Several flexibility dimensions exist within StarFabric, and as engineers begin their next platform designs, they must consider some design tradeoffs. For example, they're focusing on using transparent and nontransparent bridging functionality, and pure address routing versus path routing for communicating across the fabric. Both areas are enabled by the dual-function nature of the initial PCI-to-StarFabric bridge.
Legacy-Mode Applications: When employing the transparent-bridge function, the PCI-to-StarFabric bridge looks to software, drivers, existing BIOS, and operating systems as a standard PCI-to-PCI bridge. By executing this function in all PCI-to-StarFabric bridges throughout the system, present legacy software code can run without changing one bit, including drivers and configuration code. In this legacy mode, called address routing, a system is situated with one flat global-address space. All devices in the system are visible and accessible through their unique address ranges.
The tradeoff here is that the advanced features of StarFabric, such as path redundancy and QoS, can't be realized. But this transparent-bridge mode gives the system designer a fast time-to-market solution.
For the initialization process, one processing node must be established as the root. This can be accomplished either through strapping by the system designer, or by an election process at the hardware level. When a system is powered on, the hardware automatically initiates numerous procedures.
First, all nodes (StarFabric devices) attempt to synchronize with their link partners by initiating traffic. Once synchronization has been established, an enumeration "storm" occurs. This happens automatically by the silicon. At the end of this "storm," each node in the fabric is established with a unique fabric identification number and stored path from itself back to the root node.
At this point, the BIOS and operating system begin device discovery. To the software, all PCI-to-StarFabric bridges and StarFabric switches look like PCI-to-PCI bridges. In the discovery process, a system map, or a hierarchy of PCI bus segments, is returned. Figure 2 shows an example of a simple fabric topology and the resulting PCI hierarchy established by the discovery process. At this point, the normal process of device resource allocation is performed. System communication is now possible via standard PCI address methods.
Reprints
