Gen-Z Opens the Data-Center Door to Disaggregation Across the Rack and Row

How Gen-Z Enables Disaggregation of Memory and Modularity

Disaggregation in the context of computing refers to the ability of creating separate pools of computing resources (I/O devices, accelerators, storage, etc.) that can be allocated to applications or workloads as needed. In today’s data-center and edge environments, there are several ways to disaggregate I/O devices and storage, but not memory. 

Gen-Z enables disaggregation of memory to allow for memory devices and CPUs/SoCs to evolve independently, as opposed to being tightly coupled together as they are today. This is achieved by moving the memory controller into the memory device, creating a memory semantic link, and using low-latency forward error correction (FEC) (if needed) to drive high-speed differential pair signaling over a longer distance between CPUs and memory than is currently the case.

By decoupling memory controllers from systems-on-chip (SoCs) over a standard interface, Gen-Z provides modularity, interoperability, and compatibility between different memory types and SoC devices. Breaking the interlock between memory devices and CPUs/SoCs as described above allows for innovation separately across modules and faster time to market for these devices. Interoperability and compatibility are achieved through the robust and open standards released by the Gen-Z Consortium.

Modularity in this context means elements of computing being independently scalable, deployable, and interchangeable. This is important because it not only lowers the cost of adopting Gen-Z technologies, but also reduces the barrier to innovation for Gen-Z adopters. The key benefits of Gen-Z modularity include simplified memory provisioning, eliminated stranded memory, independent innovation timelines, scalability, and configurability.

Thanks to Gen-Z, any media can be connected within the same ecosystem. As a result, the Consortium identified the need for a methodology that allows a chassis to accommodate a wide range of devices with a variety of power, size, and connectivity requirements. By leveraging standard form factors and interconnects that scale, a system can be created in a modular fashion in which each compute element is based on a common building block. This gives chassis designers the ability to implement scalable systems that can accommodate the wide variety of resources supported by Gen-Z.

Gen-Z Form Factors and Connectors

The Gen-Z Mechanical Work Group needed a common interconnect for the different compute elements that Gen-Z can reach, such as memory, storage, and I/O devices. This required collaboration across the industry between Gen-Z, multiple vendors, and standards bodies to develop common pinouts and mechanical interfaces.

In addition, new mechanical form factors were needed to support this range of devices and the new features enabled by Gen-Z. These form factors required additional levels of modularity to scale power and bandwidth independently, as well as the capability to reach entirely new levels of each all while increasing bandwidth to the devices. 

The Gen-Z Consortium donated multiple form-factor and connector/cable (such as SFF-TA-1002) specifications to the SNIA SFF group to maintain those specifications and broaden adoption as protocol-agnostic interfaces. SFF specifications can be viewed and downloaded here.

Gen-Z form factors have been demonstrated by the Gen-Z Proof of Concept Working Group at industry conferences like FMS and Supercomputing. For example, the Gen-Z Micro Dev Kit from Smart Modular and IntelliProp (outlined in the previous Gen-Z Memory Fabric article) uses ZMM modules in the E3 form factor with a x16 SFF-TA-1002 connector interface.

The Gen-Z Form Factor (ZSFF) creates a scaling framework using simple functions to define the width, height, and depth of interchangeable form factors. It’s built on existing fundamental building blocks in the industry, specifically the 1U volumetric and thinnest pitch of possible enclosed PCB assemblies. It defines tolerances based on common manufacturing processes used across the industry and scales them along with normal dimensions (Fig. 1).

As previously mentioned, another focus of the work group was a high-performance and scalable connector. The Gen-Z Scalable Connector is a card-edge, high-density, discrete pin connector that supports vertical, right angle, straddle, and cabled installations. It provides high-performance signal integrity that can support line rates up to 112-GT/s PAM4. This high-performance signal integrity also enables a FEC-free, low-latency ecosystem for Gen-Z memory devices (Fig. 2).

Next, Gen-Z focused on creating the PECFF (PCIe Enclosure Compatible Form Factor) to meet the needs of Gen-Z based systems. The Gen-Z PECFF is essentially a standard PCIe CEM form factor with all of the same scalability, but on which the standard PCIe connector has been replaced with the Gen-Z connector. By doing so, it maintains chassis compatibility while making some tweaks along the way to enable better stacking in a 1U system.

Changes include card-to-card cabling in the form factor to enable broader adoption and reduction of the overall trace-length requirements by 60 mm. This allows for more channel budget to be used elsewhere in the system and/or power reduction in the host add-in card ASICs. Gen-Z also established support for 48-V ecosystems to deliver next-generation, higher-power devices while delivering all of the power through a high-power card-edge connection, eliminating the need for secondary cables (Fig. 3).

Facilitating Interoperability

Gen-Z standardized form factors enable interoperability by sharing common mechanical sizes, scaling in units of thickness, and sharing common interconnects. This promotes innovation by supporting a wide variety of components types, including processors, memory modules, FPGAs, GPUs, DSPs, I/O accelerators, network interface cards (NICs), custom ASICs, and more. The Gen-Z Media Box and GPU Box demonstrate that Gen-Z technology, connectors, form factors, and enabled software are real and ready for developers to innovate.

Summary

Disaggregation, a high-speed error free interconnect, and common form-factor modules help to reduce deployment and cost of ownership as well as upgrade flexibility. With the Gen-Z fabric, devices can be added, removed, and allocated as required by applications. All devices are peers, and through fabric management can be configured and reconfigured to optimize performance.