[Technology Report]
PCI Express: Hardware Designed With Compatibility In Mind
PCI Express is ramping up slowly, but it will quickly become the dominant peripheral interconnect.
PCI took over the desktop and has muscled its way into embedded systems. The serial PCI Express will eventually replace its parallel cousin in almost every avenue. Initially, bridge chips will provide access to a PCI Express switch fabric, but the lay of the land will change as more processors and peripheral interfaces replace PCI interfaces with PCI Express. At this point, PCI Express is more interesting to embedded developers in general.
PCI Express brings significant benefits yet raises new problems. The 2.5-GHz signaling frequency delivers high bandwidth that can be increased incrementally by adding more bidirectional lanes. Such speed is significantly higher than even the 133 MHz for PCI-X 1.0 or 66 MHz for PCI 2.1. What that means, then, is a call for new circuit-board design rules, as well as new test equipment to analyze these higher-speed systems. Most cases don't require new board and socket technology. It's primarily a matter of looking at the details along with proper preparation. For example, mid bus pads can be designed into PCI Express circuit-board runs so that it's possible to use test probes to examine a PCI Express lane.
Nader Saleh, president and CTO for Catalyst Enterprises, indicates that designers familiar with InfiniBand and Fibre Channel have had luck with PCI Express designs because they're used to dealing with gigahertz signals. The complexity of dealing with high-speed signals and the PCI Express protocol has delayed development by six to nine months while designers move up the learning curve. Fortunately, such companies as Catalyst, Tektronix, and Agilent are delivering new test equipment like Catalyst's SPX-16E Exercister, which handles x16 links. Designers used to megahertz designs must now deal with details like clock jitter.
PCI Express, unlike Advance Switching (AS) for PCI Express, makes compatibility paramount (see "Advanced Switching It Is"). An AS system can incorporate PCI Express devices using a tunneling mechanism (see "Advanced Switching For PCI Express: The Future Looks 'Fabric' Fast," Electronic Design, June 23, 2003, p. 36). But native AS is a packet-based system. PCI Express supports PCI and PCI-X by retaining a memory-mapped architecture. This means that PCI software doesn't change even though PCI Express utilizes packets just like AS.
A good example of this transparency is when PCI Express is used to connect a pair of PCI or PCI-X systems. A board is plugged into each system. One board has a PCI Express forwarding bridge, and the other has a reverse bridge. The boards are joined by a PCI Express link that uses fewer lines and can go further than a PCI/PCI-X bus. No software needs to be changed to perform transfers across this link.
IT'S IN THE SWITCH PCI Express appears to a device or host processor as a memory interface identical to PCI and PCI-X. Internally it's a packet-switched system that encapsulates memory reads and writes into packets. Reads are translated into request packets with address and size information. The response to these packets is the data requested.
Packets get routed through PCI Express switches. Data flows through the switch fabric without regard to the number of lanes joining each device. The initial connection from host to switch may be an x16 connection, while the endpoint has an x1 connection. In between, the switches may have an x4 connection.
This ability to mix and match connection bandwidth is a key advantage over PCI and PCI-X. Judicious use of switch capacity can optimize system performance and cost. Typically, a PCI Express switch will enable the grouping together of a variable number of lanes. For example, an x32 switch may allow an x16, an x8, and eight x1 connections or any similar combination.
PCI Express offers a maximum of 256 switch levels, but the practical implementation will be lower than this limit. Also, a switch typically adds 100 to 200 ns of delay. Therefore, a designer will need to determine that maximum delay between endpoints for a particular system.
Many other enhancements are included in addition to higher speeds and greater expansion options. PCI Express virtual channels allow systems to support isochronous (or time-dependent) data transfers and various quality-of-service (QoS) levels. Hot-plug support is standard, as is power management. There's also data-integrity support, although end-to-end CRC is an optional feature. Still, most PCI Express switch chips implement a full complement of features.
BRIDGING The first wave of PCI Express products incorporates a range of PCI Express bridges (Fig. 1). These bridges typically link PCI Express to PCI and PCI-X buses.
NEC Electronics Corp., PLX Technology, and Texas Instruments are some of the major vendors delivering PCI Express bridge chips. Not all bridge chips are created equal, yet most are relatively flexible in terms of support and bandwidth. The table shows various bridge configurations that can be used within a PCI Express system. Some bridge chips, like PLX Technology's PEX 8104 PCI-X-to-PCI Express bridge, support all configurations.
Bridging technology is finding its way inside chips as well. Chips for interfaces like Gigabit Ethernet and Serial ATA are already available with PCI and PCI-X interfaces. Moving forward, transparent PCI Express bridge chips on these interface chips reduce the bill of materials while quickly providing a native PCI Express chip. This allows more time to transition to a native PCI Express implementation. It's analogous to incorporating a PCI Express bridge on a PCI Express board containing a PCI/PCI-X device.
IN THE CHIPS The emergence of PCI Express in the server and workstation market are helped by the separation of the processor and peripherals via a chip set that typically includes a north bridge and south bridge. These are ideal places to put PCI Express support. Keep in mind that these bridge chips differ from PCI Express bridge chips.
Intel, Silicon Integrated Systems, and Via Technologies offer bridge chips that interface to a Pentium processor and PCI Express. An x16 connection is primarily directed at the video interface, while fewer PCI Express links handle peripheral interfaces. Less than 16 links is suitable, even with high-speed peripherals such as Ethernet and Serial ATA.
Via's PT890 and K8T890 north-bridge chip sets work with the VT8251 south bridge. All have PCI Express links. The north-bridge chips include an x16 and an x4 connection designed for video and peripherals, respectively. Via's 66-MHz, 16-bit Ultra V-Link connection joins the north and south bridge. This connection is sufficient to handle the x2 PCI Express link in the south bridge, which is primarily intended for low-speed PCI Express peripherals.
Silicon Integrated Systems' SiS656 north bridge links a Pentium 4 to an x16 PCI Express link. It's designed to work with the SiS965 south bridge, which contains an x1 PCI Express link along with a typical collection of built-in peripherals.
Advanced Micro Devices' HyperTransport-based Opteron and Athlon 64 will typically interface to PCI Express via a HyperTransport-to-PCI Express bridge (e.g., the SiS756 with its x16 PCI Express link and support for an SiS965 south bridge). A number of devices will interface directly to HyperTransport, such as video and high-speed communication interfaces like InfiniBand and Gigabit Ethernet. Besides, using this type of PCI Express bridge is no different than using a north or south bridge.
Reprints
