In-Depth

Storage Virtualization Pain Points

Depleted resources, storage provisioning, virtual machine sprawl and excessive virtual hard disk files are some of the key challenges facing storage virtualization users.

• By Brien Posey
• 05/01/2011

Over the last several years, storage virtualization has developed into an essential technology for organizations wishing to take advantage of server virtualization. Storage virtualization aggregates storage resources, often from multiple vendors, into a centralized resource pool that can be used by virtual machines (VMs). More importantly, though, storage virtualization facilitates fault tolerance for host servers through host-level clustering. In spite of the many benefits associated with storage virtualization, there are also numerous issues that must be overcome. If left unchecked, these issues can quickly become pain points for administrators charged with managing virtualized storage. In this article, I'll talk about some of these pain points and how you can work around them.

Depleted Resources
By far the issue that plagues storage administrators more than any other is depleted resources. I'm not talking about running out of disk space, although that can be an issue too. I'm talking about bandwidth depletion. Bandwidth depletion occurs when your data pipes are insufficient to handle the volume of data that's being sent through them. The end result is that virtual servers running I/O-intensive applications get bogged down because of a transport-layer bottleneck.

One of the key steps in addressing the issue of depleted bandwidth is accepting the idea that some types of storage cost more than others (I'll revisit this theme later on). You can look at a storage pool's cost in terms of actual dollars or in terms of bandwidth consumption. In either case, it's a good idea to reserve high-cost storage for situations in which it will be the most beneficial, and use lower-cost storage for everything else.

So what does this concept have to do with storage bandwidth depletion? Well, as you place virtual hard disk (VHD) files and application data into your storage pool, you're also creating storage I/O, which in turn adds to the storage bandwidth congestion. You can relieve some of this congestion by decreasing the storage I/O requirements, and one way to accomplish this is by looking for things you can move to less-expensive storage without negatively impacting the performance or availability of the VM.

If you have virtual Windows servers that aren't part of a failover cluster, one way you can reduce the storage I/O requirements is by moving each VM's pagefile. Microsoft has always recommended placing the pagefile on a dedicated disk if you want to achieve the best performance. This advice doesn't change just because you've virtualized your servers. Furthermore, there's no reason why the dedicated VHD has to reside within your virtualized storage pool (unless the host machine is part of a failover cluster). You can instead move each VM's pagefile to local storage on the host server. This should help reduce the amount of volume of I/O for your storage pool.

As I mentioned earlier, if you're using host-level failover clustering, offloading a VM's pagefile to local storage isn't really an option. All of a VM's VHDs must reside within a shared storage pool so that other cluster nodes can have access to them in the event of a failover. Even so, there's a way to help reduce the overall storage I/O requirements.

Take a look at how much memory each VM is using and allocate additional RAM if necessary. Pagefile swapping is I/O-intensive, so you should prevent it if you possibly can. The only way to prevent pagefile swapping is to ensure that your virtual servers have sufficient physical memory allocated to them so the OS doesn't have to perform any page swapping.

Of course, minimizing storage I/O will only go so far toward helping to ensure adequate performance. You must also ensure that your hardware is up to the job. This means provisioning your virtual storage pool with high-speed arrays and using a high-speed networking architecture such as 10GB Ethernet or Fibre Channel for connectivity between host servers and the storage pool.

Storage Provisioning
Earlier I explained that it's important to use your higher-cost storage where it really counts. So far I've talked about high-priced storage from the perspective of reducing the I/O burden of a shared storage pool. However, you must also take into account the cost of the storage itself. Your shared storage pool is expensive and has a finite amount of space available, so it's important to use that space wisely.

There are several different things you can do to make the best possible use of storage resources. One option is taking advantage of thin provisioning. The basic idea behind thin provisioning is creating a VHD file without actually consuming the amount of space you've allocated.

One of the best examples of thin provisioning can be found in Microsoft Hyper-V. When you use the Hyper-V Manager to create a VHD, Hyper-V creates a dynamically expanding VHD by default -- unless you specifically instruct Hyper-V to create a fixed-size VHD. Dynamically expanding VHDs use thin provisioning.

When you initially create a dynamically expanding VHD, it consumes less than a gigabyte of disk space regardless of how large you've told Windows to make it. As you add data to the VHD, the file size grows to accommodate the data. The biggest advantage of using thin provisioning is that it allows you to make more efficient use of your storage pool because you don't have large VHD files filled with empty space. Instead, you're only consuming the storage that's actually needed.

In spite of the obvious benefits of using thin provisioning, there are some negative aspects, too. First, dynamically expanding VHDs are not as good a performance choice as going with fixed-length VHDs, especially if multiple VHD files are stored on a common volume. This is because the expansion process that occurs as the disk accumulates data can lead to fragmentation at the physical disk level, and the performance you incur as a result of disk fragmentation may not be enough to offset the benefit of making more efficient use of your storage pool.

With thin provisioning, it's critically important to monitor your disk space consumption. After all, there's nothing stopping you from over-committing your storage pool.

Imagine, for example, that you have a shared storage pool with 1TB of shared storage (this is unrealistically small, but I'm trying to make a point). Now suppose you use thin provisioning to create three VHDs, each of which is 500GB in size. In doing so you've initially only consumed a couple of gigabytes or less of physical disk space, but you've also over-provisioned your storage pool by 50 percent.

As you add data to your VHDs, the underlying storage requirements will increase until the storage pool is completely consumed. The OSes running on the VMs are oblivious to the fact that the storage pool has been over-committed, so you won't be able to depend on your virtual servers to provide a low-disk-space warning. Your only option for preventing the storage pool from running out of disk space is to constantly monitor the physical disk consumption.

Excessive VHD Files
In the previous section I used a small virtual storage pool as a way of illustrating just how easy it can be to accidentally run your storage pool out of disk space. In the real world things aren't usually so simple. A shared storage pool may contain hundreds or even thousands of VHD files, and keeping track of all of those files can be tough. There are several different factors that can make it difficult to keep track of VHD files. One is VM sprawl. It's so easy to create VMs that in some organizations the process has gotten completely out of hand and chaos is the order of the day. In such environments, it's common for VMs that are no longer in use to continue consuming space within the storage pool.

VM sprawl isn't a problem in every organization; many organizations are conscientious about deleting VMs once they're no longer needed. Depending on what server virtualization platform is being used, however, simply deleting unwanted VMs may not be sufficient. Some virtualization products leave behind orphaned VHD files when a VM is deleted.

What can you do to stop the proliferation of unnecessary VHD files within your storage pool? The first thing is getting VM sprawl under control. Rather than allowing virtual servers to be created on a whim, adopt a formalized process through which new VMs can be requested. Requiring everyone to submit a formal written request prior to creating a VM can go a long way toward helping prevent VM sprawl.

Next, adopt a lifecycle-management program for VMs. You can even integrate such a program into the VM requisition process by requiring employees to specify how long each VM will be needed. Another option is following up with the owners of each VM every six months to determine whether their VMs are still being used.

While you're at it, I recommend taking some time to hunt for orphaned VHDs. Take an inventory of all of the VHD files in your storage pool and then begin matching those files to your VM inventory. If you find VHDs that can't be matched to a VM, those files are most likely orphans -- deleting them might reclaim some space.

Finally, remember the difference between low-cost and expensive storage. Many organizations keep gold disk images on hand that can be used in the creation of new VMs. There's typically no reason why gold images need to reside within your storage pool. You can reclaim some space within the pool by moving these gold disk images to a less-expensive storage mechanism.

As you can see, storage virtualization isn't without its headaches. Thankfully, there are workarounds for all of the issues I've discussed. These workarounds require you to be actively involved in storage management; be aware that this may tax the resources of smaller companies with fewer IT employees. One way or the other, the key to storage management is staying on top of changes by monitoring systems closely and solving problems before they occur.