The Diablo Technologies TerraDIMM puts multi-level cell (MLC) flash storage on standard DDR3 memory channels (Fig. 1). Today’s microprocessors like Intel’s Core i7 and Xeon have multiple memory channels. Flash storage has pushed the interface bandwidth on SAS/SATA and even PCI Express. Connecting flash memory directly to memory controllers provides a Memory Channel Storage architecture that has the widest, fastest interface possible without changing the processor architecture.
The TerraDIMM differs from Viking Technology’s non-volatile DRAM, the ArxCis-NV (see “The Fundamentals Of Flash Memory Storage”). The ArxCiS-NV is essentially DRAM whose contents will survive a power outage. The dual-inline memory module (DIMM) operates as a DDR3 module after the system powers up, and the contents are saved when the system powers down. The flash storage is only utilized when the power is down. The Viking approach uses an ASIC to handle the backup and restore operation. It also lets the DRAM operate like DRAM while the system is functioning.
The TerraDIMM includes an ASIC as well. It has on-chip RAM for caching, and there is enough on-module capacitance to ensure that anything written to the chip and stored in RAM will be saved in flash memory. The big difference is that all write transactions are pushed to the flash storage so there is nothing special to do when power is lost.
In the short term, writes to a TerraDIMM will be on par with DRAM, but bandwidth is limited by the MLC flash storage capabilities for larger updates. Data is also read directly from flash storage. The advantage is a significantly larger capacity compared to DRAM or Viking’s approach. Modules with hundreds of gigabytes of flash storage are available. DRAM module size is a fraction of that.
A typical configuration has multiple TerraDIMMs connected to multiple memory channels that would also contain DDR3 DRAM (Fig. 2). A BIOS mod is required to recognize the TerraDIMMs and their capabilities. DRAM is typically added in a balanced fashion with multiple DIMMs in specific sockets. TerraDIMMs do not need to follow this limitation since each DIMM is managed individually. This means a system could have a single TerraDIMM or many per channel.
At this point, the TerraDIMM is accessed using one of two methods, both employing a device driver. The first is to present the storage as a device just like a conventional PCI Express flash device or a SATA/SAS device. Its block-oriented interface works with typical server operating systems like Windows and Linux.
The other approach is to take advantage of the way the TerraDIMM storage is mapped onto the processor’s virtual memory system. This requires more coordination with the applications, but it provides direct access to the storage. The application simply reads and writes to virtual memory, and the TerraDIMM ASIC handles the operations. The only difference between it and DRAM is volatility and speed.
Diablo Technologies is targeting the enterprise space where this type of channel storage architecture has major benefits. It is equally applicable to embedded applications from data recording to heavy-duty number crunching where lots of data is required and non-volatility is an advantage.
The TerraDIMM supports flash features like TRIM, but other capabilities are yet to be exploited. For example, it should be possible to boot from the DIMMs, eliminating the need for other storage than that attached to the storage channels. It should also be possible for applications to know about and manage virtual memory flash storage, but this will require changes to the operating system and applications.
Diablo Technologies’ approach provides major benefits as is. The main limitation will be the number of DIMMs that a microprocessor can support and the capacity of the DIMMs. In many instances it may still make sense to have additional flash storage available via PCI Express or other storage interfaces adding yet another tier to the memory hierarchy.
