Diablo Technologies’Memory Channel Storage(MCS) made a big splash when it was first released as TeraDIMM, and for a good reason(see “Large-Scale Flash Moves Next To The Microprocessor” at electronicdesign.com). This dual-inline memory module (DIMM)brings terabytes of flash memory to processor memory channels instead of having it connected via disk interfaces like Serial ATA (SATA), Serial Attached Storage (SAS), or even PCI Express(Fig. 1). It essentially puts the solid-state disk (SSD) next to the cores instead of one or two peripheral interfaces away from the applications.
Related Articles
- Large-Scale Flash Moves Next To The Microprocessor
- The Changing Face Of Non-Volatile Storage
- Object Oriented Disk Drive Goes Direct To Ethernet
- NVM Express Delivers PCIe SSD Access
- What's The Difference Between SATA And NVMe?
Hard-disk storage interfaces like SATA provided sufficient bandwidth to handle devices that also had very high latency levels compared to DRAM. It made sense to utilize an interface that would hide the complexity of the storage device.
Solid-state storage changed the interface equation. Its lower latency and faster transfer rate still weren’t on par with DRAM, but it was significantly faster than hard-disk storage. The interface started to be the bottleneck. Moving SSDs closer to the processor was the reason for connecting flash memory via PCI Express. PCI Express bandwidth is higher than storage interfaces like SATA and SAS because it has to handle these interfaces as well.
Key Functionality
This is why companies like Fusion-io have been delivering PCI Express-based flash memory storage products. The non-volatile memory NVM Express (NVMe) standard is one interface designed to bring flash memory storage to the processor that is built on PCI Express (see “NVM Express Delivers PCIe SSD Access” at electronicdesign.com).
This file type includes high resolution graphics and schematics when applicable.
Still, even PCI Express can limit the performance of flash memory storage. It provides a low-overhead interface, but utilizing the DRAM memory interface can reduce that overhead further. The challenge is that this interface was designed for volatile, byte-accessible memory that requires periodic refreshing.
DDR3 DIMMs also implement a demanding communication protocol for initialization and general operation. Unfortunately, simply replacing DDR3 DRAM memory chips with flash memory chips is not an option. Diablo Technologies had other ideas.
MCS plugs into the DDR3 DIMM sockets. Its Diablo Technologies controller chip implements the DDR3 protocol and interfaceswith the on-module flash memory chips,which include support found in advanced flash memory controllers such as wear leveling.
Diablo Technologies’ first partner was SMART Modular Technologies. The SMART ULLtraDIMM employs Diablo Technologies technology along with SMART’s Guardian Technology Platform architecture for flash memory storage.
The ULLtraDIMM has less than 5-µs write latency. It has a random performance of 150kread IOPS and 65kwrite IOPS. The sustained read and write performance is 1 Gbyte/s and 760 Mbytes/s,respectively. This performance scales linearly while maintaining a consistent write latency. The DIMMs are available in 200-Gbyte and 400-Gbyte versions. They use 19-nm multi-level cell (MLC) flash memory chips. And, the ULLtraDIMM can handle up to 10 full rewrites for five years.
MCS targets enterprise class servers with multiple memory sockets and multiple channels per processor (Fig. 2). The memory channels provide the highest bandwidth interface to the processor. This accelerates memory access and enables the connection of terabytes of flash storage directly to the processor.
Know Your Tradeoffs
The tradeoff between DRAM and flash storage in large installations is critical, though. Many applications need as much DRAM as possible and fill all DRAM sockets. Users need to determine the proper mix of DRAM and MCS storage. Some applications may benefit from more flash storage.
The other advantage of using the DIMM approach in a multi-processor, multicore system is that the memory channels are logically shared among all cores. This non-uniform memory architecture (NUMA) approach has different latencies for accessing storage not directly attached to the processor chip, but it is very low. It does allow any core to access any MCS DIMM, though. Most enterprise systems will take a balanced approach, but an embedded system that could partition the software might dedicate a processor chip or two for non-volatile storage chores and attach some or all of the flash memory DIMMs directly to those chips.
DRAM DIMMs must be added in a balanced fashion when using multiple memory channels. The same type of DRAM needs to be used on each channel because DRAM is accessed in parallel. On the other hand, MCS storage can be added one DIMM at a time since each DIMM is designed to be accessed individually. Different capacity MCS DIMMs can be utilized on any memory channel, although most installations are likely to use the same style DIMM within a system.
MCS DIMMs require BIOS support so they can be recognized and properly initialized. Likewise, operating system (OS) support is necessary to handle this type of flash storage. Device drivers are the easiest way to make this work. The MCS memory then can be utilized like an SSD. The only difference is the available bandwidth and reduced latency.
The other alternative is to provide direct access by the OS and applications. This requires OS modifications that Diablo Technologies is developing. It eliminates even the device driver overhead.
In some sense, programmers are relearning technology that used to be quite common when core memory was the standard. There is a difference because core memory is more akin to non-volatile DRAM . Viking Technology’s ArxCis-NV NVDIMM uses this type of approach(see “Non-Volatile DIMMs And NVMe Spice Up The Flash Memory Summit” at electronicdesign.com). This DIMM backs up and restores DRAM to on-module flash. An off-module supercap provides transition power when system power is removed either accidentally or on purpose.
The ArxCis-NV NVDIMM runs at DRAM speeds and looks like DRAM to the processor cores. The difference between it and Diablo Technologies’ MCS is capacity and speed. Mixing DRAM, NVDIMMs, and MCS DIMMs in a system would be possible and may be useful for some high-performance, high-reliability applications.
MCS is going to have a major impact on the enterprise where it is targeted, but it also will influence embedded applications
