Planning Storage for Virtualization

The more you look into server virtualization, the more you find yourself looking back at the storage for arrangement, optimization, troubleshooting and expansion capabilities.

In my current virtualization practice, I'm preparing for a big jump upward in regards to smart storage. To help me do this, I'm reading the NetApp and VMware vSphere Storage Best Practices book. This book is written by five top-notch people, including Vaughn Stewart from NetApp.

While this book is a quick read at only 124 pages, it is right to the point in so many areas. Unfortunately with virtualization, so many things revolve around the storage product in use. This is complicated by the fact that if you change products, there is a learning curve associated with getting the new product in line with your virtualization requirements.

I'm not quite done with the book, but find myself going back and forth on configuration, terms, and a new inventory of best practices.

During my prior position working with virtualization, I was a storage customer rather than the storage provider. There were many conversations on which type of storage to use, mainly SAS or SATA disk. Yet, I wasn't involved with the storage system presentation and administration of the aggregated storage, much less a fundamentally different product series to work with. 

By bringing storage and virtualization together, a lot of benefits can be realized. Storage vendors that have vCenter plug-in and pluggable storage architecture (PSA) support will allow virtualization administrators to deliver the best virtualized storage without wasting backend resources. My earlier post on  thin provisioning is just the start. Once features like snapshots, deduplication, volume cloning, RAID selection, class of disk and other critical decisions roll into the mix, you'll quickly see that storage is everything.

Anyone want to disagree with me that storage is more important than RAM or CPU? I think that it is. 

Posted by Rick Vanover on 02/23/2010 at 12:47 PM2 comments

5 Ways Virtualization Impacts Your Career

Last week, I told you five things to know about VMware certification. Now, I want to flip that around and offer five things you need to know about putting virtualization as one of your core competencies:

1. So you know virtualization, but what about storage and networking? I mentioned how these are important before, but I can't stress this enough. When it comes to architecting a robust virtualized architecture, you'll have to be a passable network and storage engineer.

2. You will have to be a licensing expert. Licensing historically has not been one of my strong points. But, virtualization has changed how I go about general infrastructure practices, from per-processor licenses with SQL server, Windows Datacenter's unlimited virtualization rights, and even USB license dongles. You'll have to deal with it, and make it work.

3. You will have to be a data protection expert. Architecting backups, providing off-site disaster recover, replication, mirroring, RPO/RTO and other core competencies will have to be part of the virtualization professional's repertoire.

4. You will have to be a database expert. I pick on databases in particular because so many topics come into play with database servers. The good news is that now there are plenty of resources that the virtualization administrator has to prepare a strong case to support databases in this environment. Here are the best resources for supporting databases in virtualization:

5. You will have to evangelize. Unfortunately, people generally don't want to change. In order to realize new IT architectures with superior benefits, mindsets have to shift to a new way of thinking. This takes patience to explain the technology to people who may have a vague understanding of a non-virtualized environment. But should you be successful at this mode and deliver a superior virtualized infrastructure, you will build political capital in your organization.

Posted by Rick Vanover on 02/18/2010 at 12:47 PM3 comments

VMXNET3 Virtual Adapter Notes

A new vSphere feature is the VMXNET3 network interface that is available to assign to a guest VM. This is one of four options available to virtual machines at version 7 (the other three being E1000, flexible and VMXNET2 enhanced).

There are a couple of key notes to using the VMXNET3 driver. The most obvious is that the guest will appear to be using a 10 Gb/s interface. While the underlying media may not actually be 10 Gb/s, the operating system will perceive it to be. This can assist in VM-to-VM traffic on the same host and port group, and this uses CPU cycles for the local traffic in lieu of the physical Ethernet media in most situations (as does VMXNET2). What it doesn't mean is that it is 10x faster than a 1 Gb/s connection. The VMware VMXNET3 whitepaper shows the gain in performance for a test situation, available in this PDF from VMware.

It's unfortunate that this driver is not the default one for a new virtual machine. There are a few reasons for this, primarily the fault tolerant (FT) virtual machine is not supported with the VMXNET3 driver, nor is it able to be used with paravirtualized SCSI drivers (PVSCSI). Check Scott Lowe's information on this topic.

Windows Server 2008 and Linux virtual machines will benefit most from using VMXNET3 due to added support of key features such as receive-side scaling (RSS). There new features with IPv6 offloading, should you be using it. There are larger transmit and receive buffer sizes with VMXNET3, which can accommodate burst-frequent and high-throughput guest VMs.

I'm going to configure my VMs for the VMXNET3 virtual network driver. I don't have any FT-required systems, nor do I use the PVSCSI drivers.

Do you see yourself using VMXNET3? Share your comments here.

Posted by Rick Vanover on 02/16/2010 at 12:47 PM8 comments

5 Things You Should Know About VMware Certification

One of the great things about the blogging and other social networking tools such as Twitter is the ability to get something quickly. Last week, I was tweeting with fellow virtualization blogger Jon Owings of 2VCPs and a Truck. It was quite simple, John just said that he didn't know what to blog about. I replied with a topic idea and Jon immediately took it. The result is this post where Jon's views of the VMware certification are spelled out.

Now that you've read Jon's list, here's my list of five things you should know about VMware certification:

1. Certification is not a substitute for experience. While this seems to be somewhat profound, the real-world experience cannot be simulated. The VMware certification does have emphasis on troubleshooting, but unfortunately this can't be used as a broad-reaching gauge of experience. In my personal certification path, I ensured I had over two years of VI3 experience before seeking the VMware Certified Professional. It follows in the spirit of my other certification efforts as well. I didn't achieve the Windows Server 2008 MCITP credential until mid-year 2009. It is important to note that as Jon is a solution provider -- I'm not. Therefore, I don't have keep up with the cutting edge of certifications.

2. Achieving a VMware certification is the single most effective rounding off of one's external presentation. I've always believed that one's marketability is a sum of one's technical experience from current or previous jobs, formal education and certification inventory. Each one is pivotal in the well-rounded IT professional. For virtualization professionals, the VMware certification creates a nice cornerstone to that model. I've even said before that virtualization has changed my life, and the VCP certification in particular has done that for me.

3. The VCP is becoming diluted. There are quite a few VCPs out there now. Between 2005 and 2007, if you had the experience plus the certification you could be very selective on opportunities. In 2010, the novelty has worn off for sure. This is even including the fact that VMware requires that a course from a VMware Education Services (or partner) be taken by the prospective VCP.

4. Hypervisors are not enough. To round out the virtualization professional (and this includes non-VMware virtualization), other core technologies such as storage and networking are critical. Traditional IT professionals could get by with having core competencies such as Windows server technologies or Linux server skills. With virtualization, you can not put enough emphasis on storage. The more you can be a storage professional with virtualization, the more you can set yourself apart from the rest of the VCPs in the field. Networking can hold the same torch, though not as much as storage.

5. Don't be intimidated by the VCP test. If you are considering the VCP credential, by all means go for it. Virtualization is one of the hottest places to be, and I'll always feel we can have one more in this party. Besides, there are also plenty of new markets for virtualization. I think there is a market for the very small business, especially as blended cloud/virtualization solutions become more popular.

Are you considering the VCP credential? Share your comments here.

Posted by Rick Vanover on 02/11/2010 at 12:47 PM6 comments

Which LUN is Which?

When it comes to administering and designing a virtualized infrastructure, storage is one of the cornerstones of a successful design. Provisioning storage is also a mixed bag of what your best practice is and it can vary greatly by the storage in use.

My private lab has a storage system with a number of iSCSI LUNs available to a number of ESXi hosts. Some of the ESXi hosts are virtual machines themselves utilizing the host as a guest backdoor. I have one LUN for my ESXi host installed on my primary server, and a number of LUNs that are allocated for test purposes. Fig. 1 shows an ESXi host (that is a guest VM) connected to the iSCSI network that shares all of these LUNs.

Visible LUNs
Figure 1. The LUNs visible to this host are shown in the storage configuration panel of a host. (Click image to view larger version.)

The question is, how do you know which LUN is which? One way is to expand the name column, as I've done in Fig. 1. This shows the name of the LUN as it is presented from the storage system. If you jump to the ESXi command line, you can enter this command to see this across all storage adapters:

esxcfg-mpath -L

This will show additional information, but is a little more cryptic to decipher the entire multipathing configuration. Fig. 2 shows this command run on the ESXi console.

Results of running esxcfg -mpath -L
Figure 2. The listing of paths has all of the information about each target. (Click image to view larger version.)

That's all fine and good, but how do we know which LUN is which? Well, that too depends on the storage system. You can easily tell them apart by size, that is if you make them different size. I'd prefer to make everything a uniform size myself. So, then how do you tell? Well in the examples below the LUN identifier number, or serial number is revealed in each entry.:


In the string above the bolded “1” indicates it is LUN 1. There is a LUN 0 (the 2 TB LUN) and two other 1024 GB LUNs, identified as LUN 2 and 3 with the same LUN identifier strings:


This can be a good spring board to roll the LUN identifier into the VMFS datastore name to avoid confusion down the road.

How do you distinguish LUNs from each other when it comes to naming the datastores? Share your comments below.

Posted by Rick Vanover on 02/09/2010 at 12:47 PM4 comments

Distributed Virtual Switch: Go for Two!

I've come to really like the distributed virtual switch feature of VMware's vSphere 4. The vNetwork Distributed Switch, as it is officially called, is a great way to standardize how hosts are provisioned for network configuration as well as ensure consistent monitoring of guest VMs in a distributed network.

In a previous post, I showed how you can reset a host if the distributed virtual switch configuration orphans it. Hosts can be orphaned through an incorrect VLAN assignment or the wrong network interface designation for the distributed virtual switch.

What I've come to feel the best way to provision the safest environment is to configure two or more distributed virtual switches. This creates a natural security zone separation between management and guest virtual machine networking. This separation at the distributed virtual switch level would also be a full separation by cables (as well as VLANs). The guest networking port groups would be a separate distributed virtual switch and use separate physical interfaces (see Fig. 1 for an example).

Distributed virtual switches, times 2
Figure 1. Two distributed virtual switches give a natural separation to follow physical media separation as well as protection against reconfiguration issues. (Click image to view larger version.)

There are quite a few questions floating around on whether the distributed virtual switch is ready for production, Virtualization expert Mike Laverick outlines a number of them in this post. My practice going forward will be use them for all roles, including service console or ESXi's vmkernel management. The natural separation of management (including vMotion) and guest networking will make at least two distributed virtual switches in most situations. In the case of a distributed virtual switch becoming misconfigured and orphaning a host, it can be reconfigured on the fly without affecting the guest networking in the event that they are all stacked on the same virtual switch.

This really ends up matching the practice done in the standard virtual switch and port group world, making troubleshooting and logical separation intuitive.

Where are you on the distributed virtual switch? How have you separated or organized them with all of the vSphere roles? Share your comments here.

Posted by Rick Vanover on 02/04/2010 at 12:47 PM2 comments

No VMware Tools Support? Reservations, Please.

I've come across some systems that for one reason or another can't run VMware Tools. This can be due to support reasons, an operating system that doesn't officially have VMware support from the version of the hypervisor or family of operating systems, an appliance that is designed to run on a physical system but doesn't offer virtualization support, or it simply won't install on that system without something else going awry. (While this does not happen frequently, usually the vendor or application owner's response is "The application isn't supported yet with virtualization" or worse, I have to explain what virtualization is.)

As an administrator, you'll know if something will or won't work as a virtual machine. Virtual machines are a wonderful way to tuck away previous systems to use as an archive or inquiry-only role. In these situations, VMware Tools is not there to provide the necessary interaction between the guest operating system and the hypervisor.

I've also had some unexplained behavior if resources are moved around on the the host due to a busy compute and memory environment. The workaround in this situation is to assign a reservation. I usually don't use reservations, as they are typically wasted compute and memory resources. But if you must, you can do it both with licensed vSphere and VI3 installations, as well as the free ESXi hypervisor (see Fig. 1).

Resource Reservations
Figure 1. The resource reservations are shown for both memory and processor categories. (Click image to view larger version.)

If you see the reservation increase (enabled by the expandable option), you know that you have allocated too little and the resource scheduler may increase what is assigned to this category. This can be done for a resource pool, an individual VM or a vApp in vSphere. Again, I only assign reservations when a situation requires it -- lack of VMware Tools is one case where it makes sense.

Do you have systems without VMware Tools? How do you plan on resource allocation beyond the base virtual machine provisioning process? Send me e-mail (preferred) or comment here.

Posted by Rick Vanover on 02/02/2010 at 12:47 PM4 comments

Thin-Provisioning: Virtualization or Storage System?

Thin-provisioning, a great feature for virtual machines, uses the disk space required instead of what is allocated. I frequently refer to the Windows pie graph that displays used and free space as a good way to explain what thin-provisioned virtual disk files “cost” on disk. All major hypervisors support it, and there is somewhat of a debate as to whether or not the storage system or virtualization engine should manage thin-provisioning.

On the other hand, many storage systems in use for virtualization environment support a thin-provisioned volume. This is effectively aggregating the same benefit across many virtual machines. In the case of a volume that is presented to a physical server directly, thin-provisioned disks can be used in that way as well. Most mainstream storage products with decent management now support thin-provisioning of virtual machine storage types, primarily VMware's vStorage VMFS and Windows NTFS file systems. NTFS is probably marginally more supported simply due to Windows having broader support. A thin-provisioned volume on the storage system effectively is VMFS- or NTFS-aware for what is going on inside. This means that it knows if a .VMDK file is using its full allocation.

The prevailing thought is to have the storage system perform thin-provisioning for virtualization volumes. There is an non-quantified amount of overhead that may be associated with a write operation that would extend the file. You can hedge this off by using the 8 MB block size on your VMFS volumes, due to its built-in efficiencies or allocating a higher performance tier of disk. For the storage system to manage thin-provisioning, you're using on the disk controllers to manage that dynamic growth with direct access to write cache. What probably makes the least sense is to thin-provision on both environments. The disk savings would be minimal, yet overhead would be increased.

How do you approach thin-provisioning? Share your comments below.

Posted by Rick Vanover on 01/27/2010 at 12:47 PM7 comments

VMware Tools Perfmon Integration

I mentioned in an earlier post that current versions of VMware Tools now include Perfmon Integration. In a Twitter discussion, I found that this tools integration has disappeared from VMware Tools installations of patched hosts.

The VMware Tools installed with the base release of ESXi, 164009, has the Perfmon integration. In the case of ESXi, a few main categories of updates have been released since the main release. Fig. 1. shows the updates since the base release of the product, including the vSphere Update 1 release in Nov. 2009.

vSphere ESXi Updated
Figure 1. The updates for vSphere ESXi are shown since the base release. (Click image to view larger version.)

In my unscientific experiment, I determined that the VMware Tools update from 175625 release is where the Perfmon integration dropped off. These were the two updates that were selected in the Fig. 1 screen shot of the vSphere Host Update Utility. Subsequent updates that include new versions of VMware Tools will not reinstate the counters, however.

VMware is aware of this situation and they are on the topic, though only through a Twitter conversation. You can drop the Perfmon counters back into the guest operating system if you rerun the vmStatsProvider add-on for Tools. This can be downloaded from the Pivot Point blog post.

Posted by Rick Vanover on 01/25/2010 at 12:47 PM7 comments

Virtualized I/O Can Now Include Local Storage

For organizations that deal with a large amount of connectivity, managing I/O that reins in port and device costs is a complicated task. For virtualization implementations, there are a number of ways to tackle this challenge. Some organizations avert fibre channel by focusing on Ethernet-based storage protocols such as iSCSI and NFS. Another strategy is to use blade servers to consolidate servers to a chassis that can have in-chassis switching and storage I/O delivered via a limited number of ports.

In spite of this, we still see a large number of virtualization implementations using rack-mount servers in favor of blades and fibre channel storage in lieu of an Ethernet-based storage protocol. In these situations, organizations can consider consolidating I/O to cut costs in central switching (ports) and achieve better utilization for this connectivity.

Recently, I reviewed parts of the Virtensys virtualized I/O solution. I’ve mentioned virtualized I/O on this blog before after having seen the Xsigo solution as well. The Virtensys solution delivers virtualized I/O differently. For one, Infiniband is not used to deliver the virtualized I/O. Instead, standard host bus adapters, network interfaces and other PCI-Express local devices can be used as the server endpoint device to receive the virtualized I/O.

The other distinguishing feature is that local storage can be managed with virtual I/O. Specifically, the storage that is local or direct attached on a server can be presented back to the Virtensys I/O virtualization engine. Fibre channel storage as well can be delivered through the virtualized I/O. The new VIO 4008 controller concurrently virtualizes Ethernet, fibre channel and local storage.

The typical implementation is a top-of-rack solution, where each server in the rack connects to the I/O virtualization switch for the networking and storage connectivity. The Virtensys technology has a limited compatibility list, but that surely will increase over time.

Virtualized I/O can appeal to certain situations, typically driven by port cost. Virtualized I/O can raise a lot of questions in security circles, however. Frequently the debate of whether or not Layer-2 separation is "good enough" is amplified in this situation as the same physical media could potentially transport multiple security zones of network traffic while simultaneously transporting multiple integrity zones of data.

What is your take on virtualized I/O? Share your comments here.

Posted by Rick Vanover on 01/21/2010 at 12:47 PM4 comments

Distributed Virtual Switch Reconfiguration Notes

In "Fat-Finger Hero," I mentioned how recovering from a bad configuration can be an admin's saving grace. I mentioned how one of the more common areas to get stuck is host networking, and I mentioned some tools that can help you protect against configuration issues.

A new feature of vSphere, the distributed virtual switch, is one that's a perfect solution until something goes awry. Anything can go wrong, from a VLAN not being mapped to a port you are expecting, or possibly that dreaded fat finger. In one of my lab environments, I was configuring the distributed virtual switch feature and hit a snag.

I had performed the migration functionality for the hosts to go to the distributed virtual switch. This moves the management network and vmkernel communication to the distributed virtual switch from standard vswitch configuration. If the distributed virtual switch is configured with all of your expected VLANs, it may look something like Fig. 1.

distributed virtual switch
Figure 1. The distributed virtual switch feature is configured in the vSphere Client and deployed to selected hosts. (Click image to view larger version.)

Should you go 'all in' and migrate the host and its management network to the distributed virtual switch, normal migrations should have minimal interruption in communication. But if anything goes awry in the configuration, the host may become orphaned.

We've all heard of orphaned VMs, but an orphaned host? Yes, it can happen if the VLANs are not correct on the physical switch from which you are migrating. This is especially important if you are changing vmnic assignments in the process.

restore standard switch
Figure 2. The restore standard switch allows a distributed virtual switch configuration to be undone on a host. (Click image to view larger version.)

ESXi provides a nice little back-out feature for a distributed virtual switch configuration. From the main ESXi menu, press F2 to enter the customization tool. If a distributed virtual switch is configured, the "Restore Standard Switch" option is enabled. This will let you configure a standard network configuration, select an interface for the vmkernel management interface, and specify a VLAN if necessary (see Fig. 2). Be ware that it will also cause an interruption to guest virtual machines, so it should be done only on an as-needed basis.

Posted by Rick Vanover on 01/19/2010 at 12:47 PM0 comments

Fat-Finger Hero

There is risk when it comes to configuring the various items that make up a virtualized infrastructure. What do you do when things go wrong? Ideally, there are protections in place to prevent configuration errors from causing you extra work or, even worse, a new install of a host. Here are a few things you can do to protect ahead of the curve for VMware environments:

Scripted configuration: For port group and standard virtual switch configuration, the esxcfg-vswitch command will allow you repeat configuration on many hosts.

Host Profiles: For vSphere environments, host management can be managed through configuration on a number of hosts. This configuration can configure storage, networking, security settings and other aspects of ESX and ESXi hosts.

VI3 networking configuration: The ITQ VLAN and port group manager tool is used to configure networking in VI3 environments.

But what about other situations when the inevitable fat-finger entry occurs? One of the most common things that I have fat-fingered is a VLAN configuration. For things like virtual machine port groups and vmkernel interfaces, this is usually an easy correction when things are determined to be corrected improperly. It gets more complicated with the service console VLAN, IP address or other configuration that prevents you from accessing the system occurs.

Here is an example: I configured the service console of an ESX server with an incorrect VLAN, yet with the correct IP address. While it is a good idea to separate roles with VLANs, I need to use the right VLAN! A way around this without rebuilding the host (as in most cases you can't access it with this configuration) is to create a virtual machine port group on the incorrect VLAN. On that port group, then place a virtual machine and configure its network configuration to be on that IP network that the 'orphaned' host is located. Then, use that virtual machine to run the vSphere client to directly configure the service console correctly (change the VLAN to the correct entry). At that point, the host will resume its correct place on the intended VLAN. This can be especially helpful if you are not on site and have an orphaned host that needs to be reconfigured.

How have you recovered from misconfigured elements of your virtual environment? Share your comments here.

Posted by Rick Vanover on 01/14/2010 at 12:47 PM0 comments

Virtualization Review

Sign up for our newsletter.

I agree to this site's Privacy Policy.