When it comes to administering and designing a virtualized infrastructure, storage is one of the cornerstones of a successful design. Provisioning storage is also a mixed bag of what your best practice is and it can vary greatly by the storage in use.
My private lab has a storage system with a number of iSCSI LUNs available to a number of ESXi hosts. Some of the ESXi hosts are virtual machines themselves utilizing the host as a guest backdoor. I have one LUN for my ESXi host installed on my primary server, and a number of LUNs that are allocated for test purposes. Fig. 1 shows an ESXi host (that is a guest VM) connected to the iSCSI network that shares all of these LUNs.
|Figure 1. The LUNs visible to this host are shown in the storage configuration panel of a host. (Click image to view larger version.) |
The question is, how do you know which LUN is which? One way is to expand the name column, as I've done in Fig. 1. This shows the name of the LUN as it is presented from the storage system. If you jump to the ESXi command line, you can enter this command to see this across all storage adapters:
This will show additional information, but is a little more cryptic to decipher the entire multipathing configuration. Fig. 2 shows this command run on the ESXi console.
|Figure 2. The listing of paths has all of the information about each target. (Click image to view larger version.) |
That's all fine and good, but how do we know which LUN is which? Well, that too depends on the storage system. You can easily tell them apart by size, that is if you make them different size. I'd prefer to make everything a uniform size myself. So, then how do you tell? Well in the examples below the LUN identifier number, or serial number is revealed in each entry.:
In the string above the bolded “1” indicates it is LUN 1. There is a LUN 0 (the 2 TB LUN) and two other 1024 GB LUNs, identified as LUN 2 and 3 with the same LUN identifier strings:
This can be a good spring board to roll the LUN identifier into the VMFS datastore name to avoid confusion down the road.
How do you distinguish LUNs from each other when it comes to naming the datastores? Share your comments below.
Posted by Rick Vanover on 02/09/2010 at 12:47 PM4 comments
I've come to really like the distributed virtual switch feature of VMware's vSphere 4. The vNetwork Distributed Switch, as it is officially called, is a great way to standardize how hosts are provisioned for network configuration as well as ensure consistent monitoring of guest VMs in a distributed network.
In a previous post, I showed how you can reset a host if the distributed virtual switch configuration orphans it. Hosts can be orphaned through an incorrect VLAN assignment or the wrong network interface designation for the distributed virtual switch.
What I've come to feel the best way to provision the safest environment is to configure two or more distributed virtual switches. This creates a natural security zone separation between management and guest virtual machine networking. This separation at the distributed virtual switch level would also be a full separation by cables (as well as VLANs). The guest networking port groups would be a separate distributed virtual switch and use separate physical interfaces (see Fig. 1 for an example).
|Figure 1. Two distributed virtual switches give a natural separation to follow physical media separation as well as protection against reconfiguration issues. (Click image to view larger version.)
There are quite a few questions floating around on whether the distributed virtual switch is ready for production, Virtualization expert Mike Laverick outlines a number of them in this post. My practice going forward will be use them for all roles, including service console or ESXi's vmkernel management. The natural separation of management (including vMotion) and guest networking will make at least two distributed virtual switches in most situations. In the case of a distributed virtual switch becoming misconfigured and orphaning a host, it can be reconfigured on the fly without affecting the guest networking in the event that they are all stacked on the same virtual switch.
This really ends up matching the practice done in the standard virtual switch and port group world, making troubleshooting and logical separation intuitive.
Where are you on the distributed virtual switch? How have you separated or organized them with all of the vSphere roles? Share your comments here.
Posted by Rick Vanover on 02/04/2010 at 12:47 PM2 comments
I've come across some systems that for one reason or another can't run VMware Tools. This can be due to support reasons, an operating system that doesn't officially have VMware support from the version of the hypervisor or family of operating systems, an appliance that is designed to run on a physical system but doesn't offer virtualization support, or it simply won't install on that system without something else going awry. (While this does not happen frequently, usually the vendor or application owner's response is "The application isn't supported yet with virtualization" or worse, I have to explain what virtualization is.)
As an administrator, you'll know if something will or won't work as a virtual machine. Virtual machines are a wonderful way to tuck away previous systems to use as an archive or inquiry-only role. In these situations, VMware Tools is not there to provide the necessary interaction between the guest operating system and the hypervisor.
I've also had some unexplained behavior if resources are moved around on the the host due to a busy compute and memory environment. The workaround in this situation is to assign a reservation. I usually don't use reservations, as they are typically wasted compute and memory resources. But if you must, you can do it both with licensed vSphere and VI3 installations, as well as the free ESXi hypervisor (see Fig. 1).
|Figure 1. The resource reservations are shown for both memory and processor categories. (Click image to view larger version.)
If you see the reservation increase (enabled by the expandable option), you know that you have allocated too little and the resource scheduler may increase what is assigned to this category. This can be done for a resource pool, an individual VM or a vApp in vSphere. Again, I only assign reservations when a situation requires it -- lack of VMware Tools is one case where it makes sense.
Do you have systems without VMware Tools? How do you plan on resource allocation beyond the base virtual machine provisioning process? Send me e-mail (preferred) or comment here.
Posted by Rick Vanover on 02/02/2010 at 12:47 PM4 comments
Thin-provisioning, a great feature for virtual machines, uses the disk space required instead of what is allocated. I frequently refer to the Windows pie graph that displays used and free space as a good way to explain what thin-provisioned virtual disk files “cost” on disk. All major hypervisors support it, and there is somewhat of a debate as to whether or not the storage system or virtualization engine should manage thin-provisioning.
On the other hand, many storage systems in use for virtualization environment support a thin-provisioned volume. This is effectively aggregating the same benefit across many virtual machines. In the case of a volume that is presented to a physical server directly, thin-provisioned disks can be used in that way as well. Most mainstream storage products with decent management now support thin-provisioning of virtual machine storage types, primarily VMware's vStorage VMFS and Windows NTFS file systems. NTFS is probably marginally more supported simply due to Windows having broader support. A thin-provisioned volume on the storage system effectively is VMFS- or NTFS-aware for what is going on inside. This means that it knows if a .VMDK file is using its full allocation.
The prevailing thought is to have the storage system perform thin-provisioning for virtualization volumes. There is an non-quantified amount of overhead that may be associated with a write operation that would extend the file. You can hedge this off by using the 8 MB block size on your VMFS volumes, due to its built-in efficiencies or allocating a higher performance tier of disk. For the storage system to manage thin-provisioning, you're using on the disk controllers to manage that dynamic growth with direct access to write cache. What probably makes the least sense is to thin-provision on both environments. The disk savings would be minimal, yet overhead would be increased.
How do you approach thin-provisioning? Share your comments below.
Posted by Rick Vanover on 01/27/2010 at 12:47 PM7 comments
I mentioned in an earlier post that current versions of VMware Tools now include Perfmon Integration. In a Twitter discussion, I found that this tools integration has disappeared from VMware Tools installations of patched hosts.
The VMware Tools installed with the base release of ESXi, 164009, has the Perfmon integration. In the case of ESXi, a few main categories of updates have been released since the main release. Fig. 1. shows the updates since the base release of the product, including the vSphere Update 1 release in Nov. 2009.
|Figure 1. The updates for vSphere ESXi are shown since the base release. (Click image to view larger version.)
In my unscientific experiment, I determined that the VMware Tools update from 175625 release is where the Perfmon integration dropped off. These were the two updates that were selected in the Fig. 1 screen shot of the vSphere Host Update Utility. Subsequent updates that include new versions of VMware Tools will not reinstate the counters, however.
VMware is aware of this situation and they are on the topic, though only through a Twitter conversation. You can drop the Perfmon counters back into the guest operating system if you rerun the vmStatsProvider add-on for Tools. This can be downloaded from the Pivot Point blog post.
Posted by Rick Vanover on 01/25/2010 at 12:47 PM7 comments
For organizations that deal with a large amount of connectivity, managing I/O that reins in port and device costs is a complicated task. For virtualization implementations, there are a number of ways to tackle this challenge. Some organizations avert fibre channel by focusing on Ethernet-based storage protocols such as iSCSI and NFS. Another strategy is to use blade servers to consolidate servers to a chassis that can have in-chassis switching and storage I/O delivered via a limited number of ports.
In spite of this, we still see a large number of virtualization implementations using rack-mount servers in favor of blades and fibre channel storage in lieu of an Ethernet-based storage protocol. In these situations, organizations can consider consolidating I/O to cut costs in central switching (ports) and achieve better utilization for this connectivity.
Recently, I reviewed parts of the Virtensys virtualized I/O solution. I’ve mentioned virtualized I/O on this blog before after having seen the Xsigo solution as well. The Virtensys solution delivers virtualized I/O differently. For one, Infiniband is not used to deliver the virtualized I/O. Instead, standard host bus adapters, network interfaces and other PCI-Express local devices can be used as the server endpoint device to receive the virtualized I/O.
The other distinguishing feature is that local storage can be managed with virtual I/O. Specifically, the storage that is local or direct attached on a server can be presented back to the Virtensys I/O virtualization engine. Fibre channel storage as well can be delivered through the virtualized I/O. The new VIO 4008 controller concurrently virtualizes Ethernet, fibre channel and local storage.
The typical implementation is a top-of-rack solution, where each server in the rack connects to the I/O virtualization switch for the networking and storage connectivity. The Virtensys technology has a limited compatibility list, but that surely will increase over time.
Virtualized I/O can appeal to certain situations, typically driven by port cost. Virtualized I/O can raise a lot of questions in security circles, however. Frequently the debate of whether or not Layer-2 separation is "good enough" is amplified in this situation as the same physical media could potentially transport multiple security zones of network traffic while simultaneously transporting multiple integrity zones of data.
What is your take on virtualized I/O? Share your comments here.
Posted by Rick Vanover on 01/21/2010 at 12:47 PM4 comments
In "Fat-Finger Hero
," I mentioned how recovering from a bad configuration can be an admin's saving grace. I mentioned how one of the more common areas to get stuck is host networking, and I mentioned some tools that can help you protect against configuration issues.
A new feature of vSphere, the distributed virtual switch, is one that's a perfect solution until something goes awry. Anything can go wrong, from a VLAN not being mapped to a port you are expecting, or possibly that dreaded fat finger. In one of my lab environments, I was configuring the distributed virtual switch feature and hit a snag.
I had performed the migration functionality for the hosts to go to the distributed virtual switch. This moves the management network and vmkernel communication to the distributed virtual switch from standard vswitch configuration. If the distributed virtual switch is configured with all of your expected VLANs, it may look something like Fig. 1.
|Figure 1. The distributed virtual switch feature is configured in the vSphere Client and deployed to selected hosts. (Click image to view larger version.)
Should you go 'all in' and migrate the host and its management network to the distributed virtual switch, normal migrations should have minimal interruption in communication. But if anything goes awry in the configuration, the host may become orphaned.
We've all heard of orphaned VMs, but an orphaned host? Yes, it can happen if the VLANs are not correct on the physical switch from which you are migrating. This is especially important if you are changing vmnic assignments in the process.
|Figure 2. The restore standard switch allows a distributed virtual switch configuration to be undone on a host. (Click image to view larger version.)
ESXi provides a nice little back-out feature for a distributed virtual switch configuration. From the main ESXi menu, press F2 to enter the customization tool. If a distributed virtual switch is configured, the "Restore Standard Switch" option is enabled. This will let you configure a standard network configuration, select an interface for the vmkernel management interface, and specify a VLAN if necessary (see Fig. 2). Be ware that it will also cause an interruption to guest virtual machines, so it should be done only on an as-needed basis.
Posted by Rick Vanover on 01/19/2010 at 12:47 PM0 comments
There is risk when it comes to configuring the various items that make up a virtualized infrastructure. What do you do when things go wrong? Ideally, there are protections in place to prevent configuration errors from causing you extra work or, even worse, a new install of a host. Here are a few things you can do to protect ahead of the curve for VMware environments:
Scripted configuration: For port group and standard virtual switch configuration, the esxcfg-vswitch command will allow you repeat configuration on many hosts.
Host Profiles: For vSphere environments, host management can be managed through configuration on a number of hosts. This configuration can configure storage, networking, security settings and other aspects of ESX and ESXi hosts.
VI3 networking configuration: The ITQ VLAN and port group manager tool is used to configure networking in VI3 environments.
But what about other situations when the inevitable fat-finger entry occurs? One of the most common things that I have fat-fingered is a VLAN configuration. For things like virtual machine port groups and vmkernel interfaces, this is usually an easy correction when things are determined to be corrected improperly. It gets more complicated with the service console VLAN, IP address or other configuration that prevents you from accessing the system occurs.
Here is an example: I configured the service console of an ESX server with an incorrect VLAN, yet with the correct IP address. While it is a good idea to separate roles with VLANs, I need to use the right VLAN! A way around this without rebuilding the host (as in most cases you can't access it with this configuration) is to create a virtual machine port group on the incorrect VLAN. On that port group, then place a virtual machine and configure its network configuration to be on that IP network that the 'orphaned' host is located. Then, use that virtual machine to run the vSphere client to directly configure the service console correctly (change the VLAN to the correct entry). At that point, the host will resume its correct place on the intended VLAN. This can be especially helpful if you are not on site and have an orphaned host that needs to be reconfigured.
How have you recovered from misconfigured elements of your virtual environment? Share your comments here.
Posted by Rick Vanover on 01/14/2010 at 12:47 PM0 comments
This year, I again have the honor of presenting at TechMentor in Orlando. There are six main tracks in the event: Windows Client/Server, Exchange/SharePoint/Unified Communications, remote users, security, Windows PowerShell and virtualization. For the virtualization track, I will be hosting four sessions:
Along with the other presenters, we'll be delivering 13 sessions in the virtualization track. We will be covering primarily VMware and Microsoft virtualization offerings, and strive to focus on giving you what you need to do your job. The other tracks provide a great rounding out of a TechMentor event, as we all have to do a little bit of everything else as well, it seems.
In January, I will be putting together the material for these four sessions. If you are not familiar with TechMentor, take some time to look into the program. TechMentor is a great way to get into details about topics that are relevant to your roles to get results.
The event is more accessible than other events, as you can have lunch with the presenters as well as ask questions without the burden of a staunch vendor-based influence. Don't get me wrong, as I still like VMworld, but TechMentor is a much less hectic and more cozy experience for attendees.
Do you plan on attending TechMentor? If so drop a note below and I'll make sure to find you there and say hello.
Posted by Rick Vanover on 01/12/2010 at 12:47 PM4 comments
As a blogger, it is important to give relevant material for what readers need to do their jobs and learn a new technology. That's the objective of the Everyday Virtualization blog, as I provide the topics I deal with on a daily basis for readers.
Eric Siebert is one of my colleagues in the blogosphere. Eric runs the popular VMware vLaunchpad (VLP) site. The VLP is a great aggregate collection of blogs and virtualization links. In fact, the VLP and Twitter are the two resources I use most to keep up to date on virtualization topics.
Each year, the VLP accepts votes for the favorite blogs. The Everyday Virtualization blog is on the list with the best of them. Go to the VLP voting site and cast your vote for your favorite virtualization blogs!
Posted by Rick Vanover on 01/06/2010 at 12:47 PM3 comments
Well, not exactly. This holiday season, virtualization saved the day for me in one regard. While virtualization is usually the modus operandi in the workplace and for most of my lab activities, I had an opportunity to use it for my family computing needs. My three-year old daughter received a gift that was an electronic reading device that hovers over a special book to interact with learning games. While the maker of the software is fine enough in the childhood education space, I don't exactly want this software installed on our home computing systems.
This is where virtualization came to the rescue. I installed the software product on a virtual machine and used Sun VirtualBox USB device filters to connect the reading device to the virtual machine. I've mentioned here before the VirtualBox is the Type 2 hypervisor that I use most. The VirtualBox USB device filters allow devices to be connected to the host system, but presented to the guest virtual machine (see Fig. 1).
|Figure 1.The USB device filters allow a device to be assigned from the host to the guest virtual machine. (Click image to view larger version) |
The funny thing is how I explained to my daughter and wife how this works. While I was clearly more engaged with the details, things like this can help out a number of situations. This example was not wanting occasional software installed on a system, but other use cases include operating system support. Again for the home use case, my scanner only has drivers for Windows XP. The next time I need to scan a document, I'll likely set up a filter for that device and an XP system.
Have USB device filters on VirtualBox or other hypervisors helped you out? Share your story here.
Posted by Rick Vanover on 01/05/2010 at 12:47 PM3 comments
When VMware released ESXi, ESX, and vCenter 4 Update 1; all of the base components received an update to the popular virtualization suite. One of the driving factors of the update was support for both Windows Server 2008 R2 and Windows 7 which were released recently.
For organizations who use VMware Converter Standalone Edition, there may be an issue in certain situations on newer versions of ESX and ESXi. The current version of the standalone converter is 4.0.1 which was released on May 21, 2009. Update 1 for vSphere was released after that, and the new operating systems may not show up correctly when performing certain conversions. In particular, powered-off conversions may fail with the message shown in Fig. 1.
|Figure 1. The 4.0.1 version of vCenter Converter Standalone can't process a Windows Server 2008 R2 or other newer operating system.|
The workaround is simple, just change the operating system designation to a lower-family member. In this example, changing the operating system back to the non-R2 version of Windows Server 2008 allows the conversion to proceed.
For vCenter Server Installations, the 4.1.1 version of the plug-in that was issued with Update 1 seems to have similar behavior. Converting powered-off virtual machines on an ESX or ESXi host that are a newer operating system gives an "Unable to determine guest operating system" message. The resolution there is to downward configure the operating system family to Windows Server 2008 base release as well.
There various posts of messages like this in the VMware Communities, but this resolution is quick and easy for the newer operating systems.
Posted by Rick Vanover on 01/05/2010 at 12:47 PM3 comments