Switch fabrics are not just for SANs and big networked megasystems. They can also deliver high-performance, flexible connectivity for chips and boards. That's just what RapidIO aims to do—put a switch fabric on the board or backplane
And RapidIO is on its way. It's backed by a solid base of chip, board, subsystem, and system vendors, such as Agilent, Alcatel, Altera, Cisco, Ericsson, Force Computers, IBM Microelectronics, In-tel, LSI Logic, Motorola, Mercury Computing, Nortel, QuickLogic, Sky Computers, Tektronix, Teradyne, Tundra, VMEtro, and Xilinx. The RapidIO 1.1 specification is out, as is a bus-functional mod-el from Motorola. RapidIO first silicon and IP should start appearing early next year, with deployment to follow by the end of the year.
RapidIO clocks in at 125 MHz to 1 GHz with double-edge clocking on a source-synchronous 8- or 16-bit port. These ports are bidirectional (full duplex) and deliver peak data rates approaching 4 Gbytes/s for 8-bit and 8 Gbytes/s for 16-bit versions. Additionally, a serial version is being defined, which builds on existing InfiniBand/10-Gbit/s-Ethernet transceivers. This version should scale well, taking advantage of faster silicon and multiple serial connections.
Architecture The RapidIO switch fabric, unlike InfiniBand, targets real-time embedded systems. It's designed to deploy switch-fabric connectivity for chips and boards. Processor, memory controller, and controller chips can have on-chip RapidIO ports for direct connections to the RapidIO fabric. For real-time operation, RapidIO supports:- Reliable packet delivery
- Low-overhead packets for real-time operation
- Atomic Operations for coordination
- Cache coherency for shared, distributed memory
- Scalable switch fabric (64k nodes for 8-bit port)
- Two memory models (one for shared distributed coherent memory).
The basic operation visible to the software is a message. This message breaks down into RapidIO packets, up to 16 per message. These packets were designed for low-overhead, hardware-controlled operation. Their data payload is up to 256 bytes per packet. A message, then, can move up to 4 kbytes of data. The largest packet size is 276 bytes (256-byte payload, 20-byte header fields).
RapidIO supports two memory models. First is a private, nonshared, noncoherent memory where memory is addressed through the controlled software packets. Second is a shared, distributed, coherent memory that is implemented in up to 16 directory-based memory domains. The individual memory controllers maintain the directories and cache coherency. If a data element is changed, the controller marks all cached versions invalid.
Through The Fabric RapidIO's basic operation is a transaction message pair, with a request and response packet. (Not all requests have a response.) This transaction pair plays out over a switch fabric of end nodes (devices) and switch nodes. The end nodes can have one or more endpoints. These endpoints are connected through the fabric by a transaction request or response in a point-to-point transaction. The request packet travels through the switch fabric to the addressed-device endpoint. If the target device must respond with information, a response packet is sent through the fabric to the transaction source. The target can accept the packet, reject the packet, or respond with data. All three generate a response packet.For reliable delivery through the fabric, RapidIO relies on a link-level protocol, which implements local hardware control. The packets are routed via internal routing tables set up during initialization and enumeration. Each packet that passes through a port-to-port link is acknowledged via a link-level "ack."
Control Symbols Key to this link-level, node-to-node control are RapidIO's control symbols, an interesting protocol innovation. RapidIO, like most switch fabrics, defines a set of fixed field packets. RapidIO designers supplemented these traditional fixed field packets with control symbols.These control symbols are special 16-bit subpackets (with a 16-bit reversed field for error check, 32 bits total) that add link-level controls. These symbols can be built into a packet, for example, as end-or-packet fields (eop). Or, they can be dynamically inserted as the packet moves through a node. Or, the control symbols can be sent independently outside of the packet. (They have a packet_ackID.) Control symbols provide an extra level of control. Moreover, each RapidIO port is full-duplex, i.e., it maintains two connections: to and from. Thus, a receiving node can send control symbols to the transmitter while the packet is in transit instead of waiting for the transmission to finish, saving valuable time.
Routing, Flow Control Routing moves packets through the fabric, switch node by switch node. These nodes support device ID routing. That is, routing is based on the ID of the target device. The RapidIO spec provides for routing tables at each node. But the details of those tables and their use is left as an "implementation detail" in the RapidIO specification. Yet the overall latency of RapidIO's Request/Response protocol depends on the efficiency of routing the request and response packets.RapidIO defines priority-based routing and flow. Each packet is assigned a priority, Prio, which ranges from 0 to 3. The highest-priority packets take precedence over lower ones. Flow control is handled by the linked node pairs. For simplicity, there is no concept of delayed or deferred packets in a link. There are three flow-control mechanisms:
- Retry
- Throttle
- Credit-based
If there's a problem with a transmitted packet, the receiving node can request a retransmission at the link level by sending a retransmit control symbol to the transmitter. It can send this when an error is detected, even as the packet is being transmitted.
If it is the receiver—say, a slower device or one without a large enough buffer space—the sending node can slow the packet transmission down. It does this by inserting wait states in the packet itself. These are idle control symbols.
Finally, RapidIO nodes can implement a credit-based flow control. In this scheme, the receiver controls the link hop. The sending node transmits only when there is sufficient buffer space to hold the packet. Up to four buffers can be implemented in a node. Each one holds one packet. The sender requests buffer status from the potential receiver using a control symbol to ask. The sender uses a control symbol to send the buffer status to the sender.
RapidIO's four-level priority mechanism for packets can be used to help in transaction ordering and for deadlock avoidance. One way to get out of a potential deadlock situation is to raise the priority of a transaction, giving it a greater chance of getting over to the next node.
Error Management Designed for reliable packet delivery, RapidIO tries to handle errors at the link level. For example, it implements extensive bit checks for packets and control symbols. There are two 16-bit CRC counts for the packet (one added for large payloads). Also, the control symbol fields are mirrored with a trailing negative image (NANDed) as a check. Any bit error causes a retransmission or error recovery. All this is done in hardware at the link level and does not impact the host software unless the error cannot be repaired.At the end-point to end-point level, RapidIO maintains watchdog timers to time-out transactions. A late transaction times-out, causing an interrupt and packet retry and recovery. Also, the RapidIO protocol implements packet-out-of-sequence detection. Packets are transmitted in order from an end node end-point. This order is maintained at the link level using the packet_ackID field.
For catastrophic errors, the nodes can notify the host with packets passed through the RapidIO maintenance port. They can inform the host of problems, bypassing the sender path. In addition, RapidIO implements a doorbell function, an in-band interrupt mechanism for reporting errors or events to the host(s).
RapidIO Trade Association
(415) 750-8263
www.RapidIO.org
See associated figure 1
See associated figure 2
| FUNCTION |
|
|
| PERFORMANCE |
|
|