As the amount of data that we send continues to escalate exponentially, network switch fabrics must be able to scale from today's tens of gigabit/s levels to the terabit levels needed for tomorrow's systems. Additionally, our dependence on networks based on Internet protocol (IP) to handle more voice and video communications and digital entertainment makes quality of service (QoS) increasingly important to ensure clear voice transmission and smooth-flowing video streams. Such guaranteed levels of service have typically only been available to users of TDM, ATM, or Sonet networks.
But now, companies are moving to packet-based networking as a lower-cost, more flexible solution, making the availability of bandwidth on each channel a key requirement. In addition to the challenge of designing a scalable system with guaranteed QoS levels, designers must craft the system to consume minimal power while squeezing it into smaller and smaller amounts of rack space.
Taking on all of these challenges in one two-chip solution, Broadcom has developed a scalable switch-fabric solution that enables packet-based systems to act like ATM/Sonet systems in their ability to guarantee services. The system can leverage multiprotocol line cards, provide circuit emulation to replace ATM/Sonet systems, and deliver IP- or MPLS-based QoS.
The first of the two devices is the BCM8332, an 80-Gbit switch chip that contains the actual switch fabric and packs 32 input ports and 32 output ports (Fig. 1). Each port contains an integrated serializer/deserializer (SERDES) that can handle a 3.125-Gbit/s, 8B/10B encoded data stream (2.5 Gbits/s of raw data). The companion BCM8320 is a bidirectional dual 10-Gbit fabric interface that also has the fabric management logic (Fig. 2). The two chips form the heart of a high-bandwidth switch that can be used in service edge routers, multiservice switches, subscriber management systems, and more.
Previous silicon solutions in the market achieved similar integration and scalability, but they couldn't ensure bandwidth for the various services delivered over the fabric. To guarantee bandwidth, Broadcom developed specialized techniques that can buffer every flow through the fabric. In comparison, other fabric architectures typically collapse the buffers among the various flows. That restricts the amount of differentiated services that can be associated with each input.
Most fabric architectures store all traffic from different input ports at a given priority level, to a particular output port, in the same priority-based queue. This limits the service differentiation that can be provided to various input ports within the same priority level. The Broadcom chip set maintains separate storage buffers for each input queue, letting customers arrange guaranteed bandwidth allocations between input and output ports. Any allocation can be set up using different sets of weights. This makes the chip set ideal for systems that need to set up service level agreements (SLAs).
The switch fabric is implemented logically as parallel switching planes that are connected to the switch interface chip through serial links. The chip set doesn't stripe the cells across the serial links, which lets the architecture handle graceful degradation of service in 2.5-Gbit/s chunks. Therefore, designers can configure the architecture for N-1 to N*2 redundancy.
One unique aspect of the chip set is its flexible bandwidth allocation. There are no bundling restrictions on the 3.125-Gbit/s SERDES links from the BCM8320 to the BCM8332s. When designing a system, designers must decide how much front-panel bandwidth the system users want. They must also take into account the necessary amount of "speedup bandwidth" both before and after a failure. This will determine how many switch cards must be available in the system. Therefore, system architects have the ultimate in flexibility of performance, redundancy, and cost tradeoffs when designing for specific system needs.
Each serial link provides the switch interface with 2.5 Gbits/s of backplane bandwidth. By selecting the appropriate number of lines to deliver the desired total bandwidth, system integrators can provision systems with just the right amount of redundancy, rather than requiring a full duplication of the fabric to provide redundancy.
Building The Fabric: A typical switch fabric system would use a BCM8320 fabric interface chip on each line card and one to 16 BCM8332 chips to form the switch fabric (Fig. 3). Beyond the BCM8320, an Ethernet line card might contain a network processor for control and a port aggregation chip or two, such as the BCM8842, that could each aggregate up to 12 1-Gbit ports into a 10-Gbit/s SPI-4 Phase 2-compatible data stream. Then the data would be sent from the aggregation chip to the network processor over the SPI-4 Phase 2 interface.
Next, the network processor on the line card would perform its management/analysis functions on the data packet and pass the data on to the BCM8320 over a CSIX-compatible (common switch interface standard) port. With two CSIX ports, the BCM8320 can be used in system architectures that employ dual network processors on a line card, or one network processor and a high-end traffic manager. Each CSIX port can run at 2.5, 5, or 10 Gbits/s. Regardless of the speed of the interface selected, any bandwidth from 2.5 to 40 Gbits/s in 2.5-Gbit/s increments can be implemented across the backplane. Interface chips employing different bandwidths can readily be mixed in the system.
Each fabric interface chip houses 16 bidirectional SERDES ports (16 input and 16 output channels) that ship the data streams to and from the switch fabric. Both of its CSIX-compatible ports support an OC-192 data stream and can tie into traffic management or network processors over a 32-, 64-, or 128-bit wide interface. The switch fabric can be implemented with one to 16 BCM8332s to deliver a scalable bandwidth of 80 Gbits/s to 1.28 Tbits/s. Each BCM8332 chip in the fabric operates independently, providing multiple switching planes to the BCM8320 fabric interfaces.
The fabric interface chip includes all queue management and flow control functions as well as cell parsing logic. One BCM8320 supports a fixed-length cell with a 15-byte header, a 1-byte trailer, and a 96- or 112-byte CSIX payload. A single BCM8320 can operate with as much as a fourfold speedup factor (40 Gbits/s per port) to handle the inefficiencies introduced by fixed-length cells and to provide additional redundancy. Each serial link carries a continuous cell during a transfer cycle. Since the fabric consists of independent switching planes, the BCM8320s must reorder the cells received from every fabric chip on egress to guarantee in-order delivery.