Take Five With Tom Fenton

5 Ways to Clear Out IOP Bottlenecks

A look at three explicit indicators and two manual tests you can perform to know if your multipath driver is causing problems to your array's capabilities.

• By Tom Fenton
• 11/18/2014

I recently was invited to review some comparative testing EMC carried out between its PowerPath/VE multipath driver and the native multipathing driver that's shipped with ESXi. The EMC test setup was straightforward: a Dell PowerEdge R710 server was attached to an XtremIO all-flash array via a Brocade DS 300 switch, and then benchmarked (using VDbench) by running a simulated database. The only variable was the multipath driver used. This testing showed an incredible 200 percent to 600 percent improvement in bandwidth and latency when using the PowerPath/VE driver.

All-flash arrays (AFAs) that can deliver hundreds of thousands of input/output operations per second (IOPs) aren't uncommon. These arrays can greatly increase the performance of applications, but only if those IOPs can be efficiently transferred.

Judging from the test results I reviewed, the multipathing driver can make a huge difference. But the question is, "How do you know if your data path is being strangled by your multipath driver?" This is a complex problem; here are five ways to determine if your multipath driver is holding back your array's capabilities:

1. The IOPs stats collected at your array are considerably less than the array's stated capacity. Make sure you're supplying enough server I/O workload to push your array's capacity.
2. Queuing delays on the server. Most native multipathing drivers use a basic round-robin scheme, which is unable to determine congestion on backed-up I/O queues. It will keep sending the I/O down the same path regardless of the queue depth.
3. Latency increases on the server. The same I/O queuing can affect response time. This is another telltale sign.
4. Run a benchmarking test on a quiesced system. Although this is perhaps the most intrusive method because you'll need to suspend production workload, it can also be the most telling method.
5. Try another driver. EMC allows you to download and use PowerPath on a trial basis for 45 days.

If you're using ESX as your hypervisor, you can check your server's statistics using the troubleshooting tool esxtop. It takes time and commitment to master esxtop, but the payoff will be a much deeper understanding of your hypervisor.

Before AFAs became commonplace, a multipath driver's inability to move data as efficiently as possible would've been masked by an array's inability to supply the data. Now you have AFAs that unmask these weaknesses and can overwhelm a multipath driver, resulting in sub-optimal performance, which can severely hobble business-critical applications.

Determining if your array is being starved can be tricky, but the advantages of using a high-performance array can be immense. AFAs aren't inexpensive, but they do offer good value for the money, if fully utilized; technologies such as PowerPath can allow you to fully utilize them and get the full benefit from your datacenter assets.

Bonus Tip!
Use the ESXtopNGC Plug-in. VMware has released an unofficial, unsupported "Fling" that allows viewing of esxtop statistics from the vCenter Web portal. You should be able to install it in less than five minutes; it makes looking at your ESXi host stats a snap.