How-To
How to Monitor and Manage NVMe Devices Using the Command Line Interface
Ramp up your productivity by ditching the Web interface.
After working with NVMe devices for about a month, I discovered that they can be configured, monitored, managed and consumed using the various Web-based clients available in vSphere 6.5. Most importantly, I learned that they can also be managed and monitored via the command line, a feature I found indispensable for diving deep into the device. This article will outline how to navigate the command-line interface (CLI) to get the most out of your NVMe device.
The NVMe device that I was working with and will refer to in this article is a Micron 800GB 9100 PRO Enterprise HHL NVMe drive, installed in a Dell PowerEdge R610 server running vSphere 6.5. The drive was formatted with a VMFS 6 filesystem and used as a datastore for this host. My primer on NVMe technology can be found here, and my article on installing the NVMe drive here. Finally, my article on using GUI tools with NVMe drives can be found here.
Esxcli NVMe Commands
vSphere 6.5 comes with various esxcli commands to manage and monitor the NVMe devices in an ESXi host. I used PuTTY to ssh into an ESXi host so I could monitor the NVMe device using esxcli commands.
To get the NVME's HBA name and its status:
esxcli nvme device list
To display all of an NVMe device's features:
esxcli nvme device get -A vmhba2
To display only the features, such as the number of queues (nq) of an NVME device (this command can also be used to set attributes for various feature):
esxcli nvme device feature nq get -A vmhba2
To set the log level for the device:
esxcli nvme driver loglevel set 1 –A vmhba2
The outputs of some of these commands are shown in Figure 1.
[Click on image for larger view.]
Figure 1. NVMe esxcli commands.
To access deep troubleshooting of an NVMe device as it displays the device's SMART log information (also see Figure 2):
esxcli nvme device log smart get -A vmhba2
[Click on image for larger view.]
Figure 2. Results of the SMART log.
You can find PCI device information for an NVMe device with the lscpi command. The "–p" or "–v" option must be used to see the devices:
lspsi –p | egrep –I nvme
To display all the scsi devices on a host and the vmhbas with which they're associated:
esxcfg-scsidevs –A
To display all the information about the device including vendor, model and firmware revision (also see Figure 3):
esxcfg-scsidevs -ld <device name>
[Click on image for larger view.]
Figure 3. Detailed SCSI device information.
To display information about the driver for the NVMe device (Figure 4):
esxcfg-module -i nvme
[Click on image for larger view.]
Figure 4. The command for information about the NVMe driver.
One of the more useful commands displays the statistics for the device and how many commands have failed (Figure 5):
esxcli storage core adapter stats get –a vmhba2
[Click on image for larger view.]
Figure 5. Getting stats for the adapter.
To list the attributes of the NVMe device (also see Figure 6):
esxcli storage core device list -d <device>
[Click on image for larger view.]
Figure 6. The command to display storage information.
Esxcli vFlash Commands
One of the use cases for an NVMe device is vSphere Flash Read Cache (vFRC). There are various esxcli commands that can be used to examine devices that are used with vFlash. Here are some examples. To display the flash devices and show if they're being used for vFlash:
esxcli storage vflash device list
To show the details of vFlash, such as the size of the cache:
esxcli storage vflash module get
To get the name of the cache which is used to get vFlash metrics:
esxcli storage vflash cache list
To display device metrics:
esxcli storage vflash cache stats get -c vfc-413278667-vfrc-test
To reset the metric count:
esxcli storage vflash cache stats reset -c vfc-413278667-vfrc-test
Using PowerCLI with NVMe Devices
vSphere 6.5 includes $esxcli commands that can be called from a PowerCLI script to obtain the values of an NVMe device. PowerCLI guru Alan Renouf was kind enough to work with me to put together a PowerCLI script for me so that I could go through all the ESXi hosts on a vCenter cluster, display the NVMe devices on them, and list the values that can be obtained from them. The script can be found on
his Web site.
Figure 7 shows the script's output.
[Click on image for larger view.]
Figure 7. PowerCLI output.
Esxtop
NVMe devices can be used with esxtop, just like any other storage device. Figure 8 shows the output after pressing "d" to show the NVMe disk adapter (VMHBA2), while Figure 9 shows the output after pressing "u" to show the disk devices (t10.NVMe).
[Click on image for larger view.]
Figure 8. The esxtop disk adaptor.
[Click on image for larger view.]
Figure 9. The esxtop disk devices.
Despite the fact that NVMe devices are powerful devices that can, for the most part, be monitored and managed by using Web-based clients, sometimes you need to use the command line to get the important information needed to fully understand these devices.
About the Author
Tom Fenton has a wealth of hands-on IT experience gained over the past 30 years in a variety of technologies, with the past 20 years focusing on virtualization and storage. He previously worked as a Technical Marketing Manager for ControlUp. He also previously worked at VMware in Staff and Senior level positions. He has also worked as a Senior Validation Engineer with The Taneja Group, where he headed the Validation Service Lab and was instrumental in starting up its vSphere Virtual Volumes practice. He's on X @vDoppler.