In-Depth
Speed Merchant: NVMe SSD Devices In the Datacenter
The high-performance interfaces are making their way into the virtual world.
The major thrust of Non-Volatile Memory Express, or NVMe, drives used to be High-Performance Computing (HPC), big data, and databases, but now businesses are starting to do interesting things with them in the virtual world. Hypervisors and storage devices are taking advantage of their incredible performance now that they have become more mainstream and affordable. They've been around for years, but have come into their own in the datacenter this year. Their cost has come down dramatically and their performance is outstanding.
NVMe Defined
NVMe is a standard interface for PCI express and SSD drives. SAS/SATA interfaces were designed decades ago for spinning disks; they did an outstanding job in this context, but were at best a mediocre interface for SSD drives. The NVMe interface, on the other hand, was designed from the ground up for use with flash. The NVMe protocol is lightweight, fast and designed to accommodate 64,000 command queues and huge queue depths. An added benefit of NVMe devices is that they actually decrease the load on the CPU, using less than half the CPU cycles per IO of SAS/SATA devices.
How fast are NVMe devices? Recent benchmark tests show that four SAS SSDs using RAID 0 can kick out about 40,000 8K random 70/30 R/W IOPs, while a single NVMe can deliver more than 160,000. That's a 400 percent improvement over not one, but four SSD devices. NVMe devices decrease latency by a factor of two over SAS/SATA SSDs. The bandwidth improvements are 3 to 10 over SAS/SATA, as NVMe can deliver over 6Gbps. Note that these are comparisons to SSD devices; HDD drives cannot hold a candle to NVMe drives.
Need for Speed
With this performance, people are starting to do very interesting things in the datacenter with NVMe devices. In a recent test, by leveraging NVMe in a VSAN deployment, VMware was able to drive 3,200,000 IOPs on a 32-node cluster. Intel, Dell, PernixData and others have devised technologies to use the NVMe drives as huge caches on servers. All major hypervisors allow NVMes as direct-attached storage to create incredibly fast file system.
All the major flash vendors have NVMe SSDs, but currently one of the hottest NVMe drives is Intel's DC P3608 series of devices. These drives are among the top performing NVMe drives generally available today. Intel's DC P3608 uses eight lanes on a single PCIe 3.0 slot, but has dual NVMe controllers. What this equates to is outstanding performance: 5GB/s of bandwidth, 850k random reads and 300k random 70/30 R/W IOPs.
The drives come with 1.6, 3.2 or 4TB of capacity. To generate these performance numbers, I've been told Intel used the standard driver and no special tuning. The P3608 has drivers for all common hypervisors: ESXi, Hyper-V, Xen and KVM. Severs are starting to support NVMe in a big way; for example, the Dell R930 can be ordered with eight NVMe devices. Storage companies have also taken notice of NVMe devices and are starting to use them for hot blocks in their storage nodes.
Replacing HDDs?
Will NVMe devices replace HDDs? Yes and no. As the price of flash devices decreases, they'll displace the 2.5" spinning disk. We're starting to see price parity between flash and fast spinning (15K), relatively low capacity HDDs. In the next year flash will start to be price competitive with 10K drives, and by 2018 experts believe that flash devices will be cost comparable with slow (5K) HDDs. Large capacity 3.5" drives will still be in the datacenter for cold and lukewarm data, but hot and warm data will live on flash.
The Target Market
Are NVMe devices for everyone? No. Only the latest generation of severs have the PCIe 3.0 slots needed to take full advantage of NVMe devices. To some, the price of NVMe devices will be prohibitive. Others will find that their workloads don't require the performance that NVMe offers. Those with current servers, a heavy workload and the budget should be able to justify the inclusion of NVMe devices.
About the Author
Tom Fenton has a wealth of hands-on IT experience gained over the past 30 years in a variety of technologies, with the past 20 years focusing on virtualization and storage. He currently works as a Technical Marketing Manager for ControlUp. He previously worked at VMware in Staff and Senior level positions. He has also worked as a Senior Validation Engineer with The Taneja Group, where he headed the Validation Service Lab and was instrumental in starting up its vSphere Virtual Volumes practice. He's on X @vDoppler.