Virtual Midplane Realizes Ultrafast Card Interconnects

The extraordinary performance of today's networked systems de-mands high-speed, point-to-point contacts for card interconnects. It has become crucial to completely rethink all aspects of traditional backplane and midplane card-interconnect technology and develop a more effective structure for this new interconnect environment. It's no longer adequate to simply promote existing solutions into an applications space that has changed radically.

The virtual midplane—a novel structure that orients cards perpendicular to each other on two sides of an imaginary plane—looks promising to meet the performance needs of networked systems both today and tomorrow. This new card-interconnect concept could provide such significant benefits as ease of migration to higher speeds, simple system upgrades, and the incorporation of all components of a system-area network within a single box. In fact, the virtual midplane could bring both ultrafast and optical card interconnects a lot closer to mainstream reality. Although this proposed technique is as yet unproven, its many benefits certainly warrant further examination.

With the virtual midplane, the card-to-card connections exist at the intersections of the cards on each side of the imaginary plane (Fig. 1a). The structure uses self-aligning connectors and a card cage with card-slot aligners that are oriented differently on the two sides of the midplane. Some advantages of this arrangement are its ability to easily reach 10 GHz or greater, simplify optical card interconnect, and potentially place components of a system-area network in a single card-interconnect structure.

In Figure 1a, various colors for different cards distinguish functions within the system. The yellow cards represent switch cards, which make all of the connections between the other cards. Clearly, this novel structure minimizes the distance a signal needs to travel between boards and the disruptive electrical environment that the signal encounters.

Performance Parameters: The two cards shown in Figure 1b offer a better illustration of how they might be connected. The green rectangle and the blue parallelogram in the two opposing planes are joined with a single gray connector. While a single connection is shown here, only available space should limit the number of connections between two cards. Note, once again, that no fixed midplane or backplane exists, but only a virtual midplane and function cards.

The arrangement of Figure 1 has many advantages. First, there's no appreciable length that the signal must travel to get from the transmitting function card to the receiving function card. Also, the signal needs only to travel over a single connector, rather than two connectors at the two ends of a midplane or backplane card. This establishes a high-performance environment.

For maximum-frequency signals, it's possible to place a discreet transmitter and receiver (or transceiver) at the connector stub. With this arrangement, the transmitted signal only has to traverse a minimum distance of millimeters. On-card, to and from this transceiver, either a high-speed serial signal or a low-speed parallel signal can be transmitted in a well-controlled electrical environment. A 10-GHz signal (or greater) could be easily transmitted. The transceivers could be located further from the stubs in performance environments that aren't possible today.

Other interesting effects of the virtual midplane concern performance. For example, an optical connection can be implemented very simply. Fiber isn't required. Instead, just a direct or free-air connection is needed between the laser on the transmitting card and the photo detector on the receiving card through an enclosed, self-aligning optical connector.

A more important performance item is system upgradability. The fixed physical medium over which the signal is transmitted and the multipoint environment of previous card-interconnect systems significantly limit performance upgrades. Because the industry is now moving away from the multipoint solution, the only remaining barrier to upgrades is the backplane itself.

The virtual structure has reduced this barrier tremendously. A completely upgradable system results, as long as the connector (or an upgraded version of the connector) can handle higher performance. Two forms of upgrade are possible in this structure. One approach would be to upgrade channel performance, with backward compatibility (i.e., 10-, 100-, 1000-Mbit/s Ethernet). Another approach is to expand the number of connections between cards.

System Functionality: Many configurations can be implemented with this structure. Figures 2 and 3 illustrate the wide array of complexities encountered and the system-level advantages that the structure offers. These two figures also give some idea of the variety of solutions available. Note that for each, the view is card-edge oriented for both planes of cards.

Figure 2 depicts a very straightforward switch solution in which all cards in both planes can easily communicate. For systems with upgrade capability, this solution is the most straightforward. Once the switches get an upgrade, any other card in the system can be upgraded as required, transforming the system into the next-generation system. This begins to look a lot like a data center in a single system with two-stage switching and redundant switches.

Yet this isn't the only possible solution when using a virtual midplane structure. As seen in Figure 3, direct connections can be used. Function cards 1 and 2 represent two different functions. In this case, though, if it's necessary to communicate between function cards in the same plane, some vehicle must be incorporated into the cards in the opposite plane.

For the example shown, the I/O cards or the memory cards could be em-ployed to communicate between function cards in the same sector. Four of the function 1 cards and four of the function 2 cards are in the same sector. Any of these eight cards could intercommunicate. In the example, function cards in different sectors have no communication path available.

Figure 4 shows how communication between cards in a switched system would proceed. Single-stage switching is used when communicating between cards in the same plane with only switched connections. Communications between cards in opposing planes using only switched connections require two-stage switching. Figure 4a shows communication between cards in the same plane. If communication is required between cards in the same plane, the path will be:

1. The source function card in the first plane communicates with the switch card in the second plane.
2. The switch card in the second plane communicates with the destination function card in the first plane.

Two-stage switching, used for cards in opposite planes (Fig. 4b), would progress as follows:

1. The source function card in the first plane communicates with the switch card in the second plane.
2. The switch card in the second plane communicates with the switch card in the first plane.
3. The switch card in the first plane communicates with the destination function card in the second plane.

While the dual switch cards in each plane are necessary for redundancy in the switch-oriented system, these cards could also be used in tandem for load sharing.

The two system-area networks of Figures 5 and 6 indicate another possible advantage. Figure 5 depicts a general description of a system-area network, or data center. Figure 6 shows how this could work in a single system using the virtual midplane approach.

All functionality of a system-area network, at least at a first level of detail, can be built into one system via the virtual midplane structure. Redundant paths, and one- and two-stage switching, are utilized throughout the example. These are all available with the interconnect structure being described and by using the switch-fabric approach of Figure 5.

Some system architects may choose to add direct connections between function cards in opposite planes to realize the shortest path between critical functions. Redundancy would still be available if the switch path re-mained. For a system with a fixed definition of function cards for each card slot, the direct function-card-to-function-card connections would be a straightforward exercise.

But for a system that needs flexible scalability (like the ability to add routers, servers, controllers, and memory, as a given application requires), it can become a complex issue that necessitates methodical planning. That's because the location of the connection between two cards varies slot by slot. Unless a connection is made at every intersection of cards, the connection location on a given card might not correspond to the correct card in the opposite plane. Or even more disconcerting, the connection on the opposing card at the expected intersection point may not exist.

An alternative for the flexible structure described here is to use only direct connections between cards, possibly at every card intersection, rather than switch cards. This solution presents its own complexities, which must be carefully studied and evaluated during system architecture design. In an extremely complex or extremely high-performance system, the solution could ultimately prove very powerful.

An example application may involve a high-end communications system. The interconnect at each card would, effectively, be a semi-mesh network, where each card would be connected to all cards in the opposite plane. This might work especially well if all traffic is always sent from one plane to the other. A switch on each card would, of course, allow for connections of all cards.

Issues: Because the virtual midplane structure is new and radically different from the norm, there are still a number of critical issues to be resolved:

• What applications are most suited for this structure?
• How can this configuration be optimized for a particular application?
• How can connectors and cages be implemented that don't require the traditional midplane or backplane printed-circuit card?
• What is the best configuration for connectors?
• How will cards be inserted?
• What is the best way to provide power distribution?
• How can adequate airflow be achieved?

Consider the airflow issue. If the card cage is arranged so one plane of cards is vertical while the other is horizontal, one would have easy access to the cards from two sides of a cabinet. But the horizontal cards would have little or no natural airflow. Horizontal fans would be necessary for one side of the card cage.

Another option would be to place the card cage so that all cards are oriented vertically, similar to the orientation in Figure 1a. Airflow may not be an issue in this case, but access to the cards from the top and the bottom would be required. Also, the size of the connectors may be too big, thus blocking the airflow. While this vertical option may be possible for a single-cage system, it would require some innovation in a rack-mount version. A card cage that has a sliding access, for instance, could be used for this configuration.

Another suggestion orients the card cage so that every card is at a 45° angle. Though somewhat restricted, airflow would be possible in this configuration.

Also, some wasted space would exist within the rack. Testing would be required to ensure adequate airflow. In the final analysis, horizontal fans for the horizontal cards and vertical fans combined with natural airflow for vertical cards may be the most straightforward solution for card access and viable airflow.

Power-distribution concerns point, once again, to the switched solution. It may be possible to distribute power from the switch cards, which would always be present in a working system. This approach also provides redundancy for the power, as well as the switching. Of course, placing a midplane card for interconnect would also allow for power distribution.

Without a doubt, the virtual midplane is a most intriguing and promising configuration for the ultimate card-to-card interconnect solution in system networking. Whether it can deliver all that it promises remains to be seen.