How to Monitor and Manage NVMe Devices Using the Command Line Interface

Ramp up your productivity by ditching the Web interface.

After working with NVMe devices for about a month, I discovered that they can be configured, monitored, managed and consumed using the various Web-based clients available in vSphere 6.5. Most importantly, I learned that they can also be managed and monitored via the command line, a feature I found indispensable for diving deep into the device. This article will outline how to navigate the command-line interface (CLI) to get the most out of your NVMe device.

The NVMe device that I was working with and will refer to in this article is a Micron 800GB 9100 PRO Enterprise HHL NVMe drive, installed in a Dell PowerEdge R610 server running vSphere 6.5. The drive was formatted with a VMFS 6 filesystem and used as a datastore for this host. My primer on NVMe technology can be found here, and my article on installing the NVMe drive here. Finally, my article on using GUI tools with NVMe drives can be found here.

Esxcli NVMe Commands vSphere 6.5 comes with various esxcli commands to manage and monitor the NVMe devices in an ESXi host. I used PuTTY to ssh into an ESXi host so I could monitor the NVMe device using esxcli commands. To get the NVME's HBA name and its status:

esxcli nvme device  list 

To display all of an NVMe device's features:

esxcli nvme device  get -A vmhba2

To display only the features, such as the number of queues (nq) of an NVME device (this command can also be used to set attributes for various feature):

esxcli nvme device feature nq get -A vmhba2 

To set the log level for the device:

esxcli nvme driver loglevel set 1 –A vmhba2

The outputs of some of these commands are shown in Figure 1.

[Click on image for larger view.] Figure 1. NVMe esxcli commands.

To access deep troubleshooting of an NVMe device as it displays the device's SMART log information (also see Figure 2):

esxcli nvme device  log smart get -A vmhba2
[Click on image for larger view.] Figure 2. Results of the SMART log.

You can find  PCI device information for an NVMe device with the lscpi command. The "–p" or "–v" option must be used to see the devices:

lspsi –p | egrep –I  nvme 

To display all the scsi devices on a host and the vmhbas with which they're associated:

esxcfg-scsidevs –A 

To display all the information about the device including vendor, model and firmware revision (also see Figure 3):

esxcfg-scsidevs -ld  <device name>
[Click on image for larger view.] Figure 3. Detailed SCSI device information.

To display information about the driver for the NVMe device (Figure 4):

esxcfg-module -i nvme
[Click on image for larger view.] Figure 4. The command for information about the NVMe driver.

One of the more useful commands displays the statistics for the device and how many commands have failed (Figure 5):

esxcli storage core adapter stats get –a vmhba2
[Click on image for larger view.] Figure 5. Getting stats for the adapter.

To list the attributes of the NVMe device (also see Figure 6):

esxcli storage core device list -d <device>
[Click on image for larger view.] Figure 6. The command to display storage information.

Esxcli vFlash Commands One of the use cases for an NVMe device is vSphere Flash Read Cache (vFRC). There are various esxcli commands that can be used to examine devices that are used with vFlash. Here are some examples. To display the flash devices and show if they're being used for vFlash:

esxcli storage vflash  device list 

To show the details of vFlash, such as the size of the cache:

esxcli storage vflash module get

To get the name of the cache which is used to get vFlash metrics:

esxcli storage vflash  cache list

To display device metrics:

esxcli storage vflash  cache stats get -c vfc-413278667-vfrc-test

To reset the metric count:

esxcli storage vflash  cache stats reset -c vfc-413278667-vfrc-test
Using PowerCLI with NVMe Devices
vSphere 6.5 includes $esxcli commands that can be called from a PowerCLI script to obtain the values of an NVMe device. PowerCLI guru Alan Renouf was kind enough to work with me to put together a PowerCLI script for me so that I could go through all the ESXi hosts on a vCenter cluster, display the NVMe devices on them, and list the values that can be obtained from them. The script can be found on his Web site. Figure 7 shows the script's output.

[Click on image for larger view.] Figure 7. PowerCLI output.

Esxtop NVMe devices can be used with esxtop, just like any other storage device. Figure 8 shows the output after pressing "d" to show the NVMe disk adapter (VMHBA2), while Figure 9 shows the output after pressing "u" to show the disk devices (t10.NVMe).

[Click on image for larger view.] Figure 8. The esxtop disk adaptor.
[Click on image for larger view.] Figure 9. The esxtop disk devices.

Despite the fact that NVMe devices are powerful devices that can, for the most part, be monitored and managed by using Web-based clients, sometimes you need to use the command line to get the important information needed to fully understand these devices.

About the Author

Tom Fenton has a wealth of hands-on IT experience gained over the past 30 years in a variety of technologies, with the past 20 years focusing on virtualization and storage. He currently works as a Technical Marketing Manager for ControlUp. He previously worked at VMware in Staff and Senior level positions. He has also worked as a Senior Validation Engineer with The Taneja Group, where he headed the Validation Service Lab and was instrumental in starting up its vSphere Virtual Volumes practice. He's on X @vDoppler.


Subscribe on YouTube