We live in interesting times when it comes to technology. That much is certain. Interesting times because so much change is being introduced to the traditional way we did computing. Virtualization revolutionized the way we do computing, and changed it forever and for better.
The most important thing that virtualization introduces is the abstraction layer. Some of you may not have thought about this profoundly, but there are several layers of abstraction, some we have already achieved and some we hope to achieve in the future. The different abstraction layers are the fundamental building blocks of what we call "cloud computing."
So what are the different layers of abstraction? Here, I have listed some:
Abstraction of hardware from software This type of abstraction is commonly known as the hypervisor and serves as an intermediary layer between hardware and the software that installs immediately on it. This type of abstraction layer is available for servers and desktops.
Abstraction of applications from operating systems This type of abstraction is more commonly known as application virtualization and occurs anytime you are able to run an application on an operating system without actually having to install it on that operating system, or have it modify that operating system in any way -- it is simply layered on top of it.
Abstraction of user workspace This is more commonly known as user workspace virtualization. In earlier days it was known as profile management. The technology and software vendors have advanced so much that it would not be fair to just label them profile managers, since the technology does so much more. With user workspace virtualization, you abstract the user settings, files and preferences from the reliance on the local operating system by virtualizing them and making them available to the user regardless of device or access method. The user consistently gets the same user experience.
Abstraction of network This is commonly known as virtual networking, which allows virtual machines to connect to virtual networks. VMs are completely abtracted from the physical network while providing seamless integration.
I/O abstraction One of my favorite topics in the IT world today is the abstraction of I/O, more commonly known as I/O virtualization or converged networking. The idea behind I/O virtualization is to standardize the transport mechanism and eliminate moving parts.
For example, if you use an Infiniband card or a 10GE card, you are capable of transporting regular Ethernet traffic, 10GE, Fiber Channel Traffic and with some technologies, even FCoE traffic. Furthermore, based on the technology you deploy, you can take this farther by carving out virtual cards that take advantage of the same transport. If you place an InfiniBand card in your server that has a 20GB connection back to the top of rack switch, you can then carve out this 20GB of bandwidth into 3x1GB Ethernet, 1x10GE etc., without changing the cabling or the card in the server. When new technology comes out tomorrow, it can easily be added to the top of rack switch and transported using the same cable. Yes, it's future-proof.
Abstraction of storage This type of abstraction is interesting and depending on who you ask, you will get a different answer. Certainly, technologies are out there that can do storage virtualization in the form of abstracting different storage platforms behind a single array and managing it that way. But the abstraction I am looking for is one that eliminates the need for LUNs, for RAID, and is true storage virtualization, where things like VMware datastores are not reliant on an underlying LUN for instance, where a VM is not tied to a LUN, etc. While this form of abstraction is not here yet, it is within reach and a matter of time.
All these abstraction layers independently are great technologies. Collectively, they're the foundation for cloud computing. Some of these technologies have existed for a while, others are new and a few are yet to be developed, but one thing is for certain: We live in the age of abstraction.
What do you think? Have I missed an abstraction layer?
Posted by Elias Khnaser on 04/19/20117 comments
When I
wrote about
type-1 client hypervisors, I made it very clear that I am biased towards them. I reiterate that again, so as to avoid getting grilled in the comments section below.
Don't get me wrong. I have VMware Fusion on my laptop and use it on a daily basis, but using a type-2 client hypervisor enterprise-wide? No thanks!
More
Posted by Elias Khnaser on 04/14/20117 comments
Recently, one of my customers was trying to decide whether or not to use type-1 or type-2 client hypervisor.
For those of that are not familiar with the technology, type-1 is a bare-metal install. You intall a thin software layer (the hypervisor) that then allows you to create virtual machines. It's very similar to VMware ESXi, Citrix XenServer and Microsoft Hyper-V, except instead of installing on server class hardware, you install it on laptops or desktops. Type-2, on the other hand, is installed on top of an operating system. An example is VMware Workstation, Fusion, Microsoft Virtual PC and Parallels.
More
Posted by Elias Khnaser on 04/12/20116 comments
When designing your vSphere 4.1 infrastructure, it is important to know how to tweak it properly in order to yield the best performance possible based on your physical infrastructure. What I mean by that is: knowing the limitations of your network will determine how many concurrent vMotions you can successfully run without impacting performance (as an example).
Now there are many articles that describe how to limit the number of concurrent vMotions on earlier versions of ESX, but I could not find one that discussed how to limit it on vSphere 4.1. So I decided it would be useful to share what I know.
vMotion, and Storage vMotion operations for that matter, carry a per-resource cost of performing that operation on a per ESX host basis. For example, with vSphere 4.1 the cost of carrying a vMotion operation is 1, while in earlier versions of vSphere like 4.1, that cost was 4 (as is the case with ESX 3.5 as well).
Consequently, the maximum number of concurrent vMotions per ESX 4.1 host is 8, while the old maximum for 4.0 is 4, and for ESX 3.5 it's also 4. Now, it is important to know the maximum vMotions shared between Storage vMotions and vMotions. The cost of a Storage vMotion in vSphere 4.1 is 4. Therefore, you can have 4 concurrent vMotions and 1 Storage vMotion on a single ESX host.
To control these value, you can log in to your vCenter server, point to Administration | vCenter Server Settings. Table 1 lists the value, its cost and its maximum as it pertains to vSphere 4.1.
| Table 1. vSphere 4.1 settings for limiting vMotions. |
Value |
Description
|
Default Cost |
Maximum |
vpxd.ResourceManager.
costPerVmotionESX41 |
Cost to perform vMotion |
8 |
8 |
vpxd.ResourceManager.
costPerSVmotionESX41 |
Cost to perform Storage vMotion |
4 |
8 |
|
|
Those settings assume a 10G Ethernet for vMotions. vSphere 4.1 adjusts this maximum based on the network interface it detects; for example, a 10G Ethernet be default allows 8 concurrent vMotions, while a 1GB Ethernet defaults to 4 concurrent vMotions. If a less-than-1 GB network is detected, it defaults down to 2 concurrent vMotions.
Another way to modify these values is to edit the vpxd.cfg file located on the vCenter server.
Posted by Elias Khnaser on 04/05/20117 comments
I got lots of feeback on my last blog, "
Desktop Virtualization vs. Physical Desktops: Which Saves More?
" Paulk from SF asked: 1. Won't there be client OS licensing costs for the virtual desktops or is that covered in the desktop virtualization software cost? 2. Does the $400/seat client operational/maintenance costs cover the operation and maintenance of the blade servers as well, or just the thin clients?
More
Posted by Elias Khnaser on 03/31/20112 comments
One of the biggest hurdles for desktop virtualization adoption is price. Through all my interactions with customers, I am always hearing: "I heard it was more expensive, I heard there are no cost savings," etc. So, let's compare a desktop virtualization rollout versus a traditional physical desktop rollout and see if it truly is more expensive.
That being said, keep in mind that from a CapEx expenditure stand point, you will not see much savings. But you will see significant OpEx savings. Usually when I say this, customers will say, "My CFO does not care about OpEx, we can quantify OpEx, we can't touch it." I say, have a little more faith. I will accept that argument and respond to you as follows: While it is not easy for every organization to quantify or justify OpEx, the next time your manager needs a project completed in a week and you have no cycles or your current employees have no cycle, the only option is to hire more help or use consultants.
More
Posted by Elias Khnaser on 03/29/201115 comments
Microsoft announced some really cool features with the upcoming System Center 2012. The demos and features are just awesome.
First you can now build private clouds and manage Citrix XenServer in addition to what was already supported (VMware ESX and Hyper-V, of course).
The cool thing about the new release is the way it aggregates physical equipment. You now have the ability to find bare metal servers, communicate with their management port and deploy an image on them. Sure, that's cool, but nothing special; we have seen this before, right? Yet, what's really cool is how you can add it to a resource pool and extend capacity on the fly. What I am very curious to know is, if you can have a cloud that is powered by heterogeneous hypervisors, for example, can resources be pooled from ESX, XenServer and Hyper-V under the same private cloud?
System Center 2012 will now be able to manage mobile devices, which is huge and reinforces a notion that I have been spreading for a long time: Mobile devices, -- cell phones, in particular -- are the next PC. System Center 2012 will allow you to manage devices built on Google Android, Apple iOS, Windows 7 Phone and Symbian.
Of course, I would not be this excited about it if it did not touch in some way on desktop virtualization, and the System Center 2012 announcement touched a lot on this subject. You will be able to optimize application delivery to your users based on the device they are using. System Center detects when someone is using an iPad and enforces the iPad policy and delivers applications and desktops that would run optimally in iPad, such as VDI for example. However, if the user is using a laptop, then it would deliver applications using App-V and would allow offline access as well. All seamless, all policy driven, all automated -- and all cool.
Microsoft also mentioned and demonstrated System Center 2012's ability to leverage a single Windows image for multiple applications -- basically, it's application and OS separation, as well as OS and endpoint device separation. Again, it's the modularization of IT and Microsoft's coming to terms with the fact that a desktop does not necessarily mean what is under your desk and that there will be instances where devices used don't necessarily have an OS.
I think System Center 2012 and all its subcomponents is going to be one of the most exciting releases Microsoft has had in a long time.
Posted by Elias Khnaser on 03/24/20115 comments
VMware PVSCSI is an optimized SCSI controller driver that is optimized to handle heavy I/O loads generated by a virtual machine. I am often asked, "When should one use VMware PVSCSI?" The answer is simple: Use it when virtualizing high I/O applications, such as Microsoft SQL Server or Exchange Server.
VMware recommends that you use the PVSCSI driver when virtual machines are generating over 2,000 IOPS. That is a very high number of IOPS. To put things in perspective for you, a 15K SAS drive generates about 170 randomized IOPS.
That being said, the LSI SCSI driver that is available and used in most cases by default is optimized to handle both low and high I/O and can adjust accordingly. In most cases, the use of the LSI SCSI driver will yield excellent performance and will consume acceptable CPU cycles on the ESX host.
Where VMware PVSCSI shines is when you get into the larger IOPS requirements. In those cases, the PVSCSI driver will consume significantly fewer CPU cycles than the LSI driver.
Now with the release of vSphere 4.1, VMware tweaked the PVSCSI driver for better performance with workloads generating less than 2,000 IOPS. However, the accepted best practice still dictates the use of the LSI driver for all low to medium workloads.
VMware spent a lot of time tweaking the LSI driver for random VM workloads and, as such, it's a victim of its own success. Now, I am sure that with every release of vSphere, the PVSCSI driver will be further enhanced and will get to a point where it can handle all types of workloads without any penalties.
Posted by Elias Khnaser on 03/22/20110 comments
There is an application virtualization riddle out there, wrapped in DVI (Desktop Virtualization Infrastructure) enigma, inside a vendor frenzy. Everyone wants to virtualize applications, and every vendor tells you to do it differently. So what is the right way of determining when to bake an application as part of the image, virtualize an application, stream it or deliver it via Terminal Services?
I will not give you the standard consultant line of "it depends," but rather I will start by saying that baking the application as part of the image should be the last resort approach -- it is the last ditch effort when all else fails. I bring this up first because I have seen many vendors recommend, off the bat, to bake it in the image. NO. OK, Eli, but saying no is easy, Why no? Well, for starters, you have now increased the size of that image exponentially, but assuming you are working off a master image technology (LinkedClones, Machine Creation, Provisioning, etc...) that is not a huge deal. Or is it? if you install all applications locally, doesn't your application updates become more frequent? Thus you have to constantly update that master image? Will you not be transferring the same issues you have on the physical desktops to the virtual desktops? Application conflicts, compatibility, testing, etc...?
More
Posted by Elias Khnaser on 03/17/20113 comments
As organizations continue on the journey of virtualizing more servers, there's a need for more attention and customization in order to yield the desired performance benefits while taking advantage of the virtualization drivers. This is especially true for those servers that host applications dubbed as tier-1 applications, such as SQL, Exchange and Citrix XenApp.
Citrix XenApp is particularly interesting when virtualized due to the nature of the server that typically consumes a lot of CPU, memory and disk I/O. So, addressing these categories in particular is imperative. Here are some best practices on how to tweak Citrix XenApp on VMware vSphere 4.1.
At the vSphere Infrastructure Level
At the virtual infrastructure level, the use of VMware vSphere 4.1 or newer with vStorageAPIs for Array Integration is imperative. VAAI offloads certain tasks previously executed in the hypervisor onto the storage array; however, the most important feature of VAAI as it pertains to virtualizing XenApp is that it offloads SCSI locking to the array. When you have multiple XenApp VMs on the same LUN and they are simultaneously utilizing the disk, the benefits of VAAI will surely be apparent.
If you have not upgraded to vSphere 4.1, I strongly recommend it, especially if you intend on virtualizing tier-1 applications. Check with your storage manufacturer to make sure your array supports VAAI. In some cases you may have to upgrade the firmware on your controller to take advantage of it.
Another tweak to the vSphere host is again related to heavy workload disk performance, and XenApp is most definitely fits that category. This tweak addresses the maximum simultaneous number of outstanding disk requests to a specific LUN.
By default, the limit is set to 32 and can be increased to a maximum of 256. This value should be set to the maximum as a regular performance best practice, and in particular when tier-1 applications are involved. This value can be changed by navigating to the Configuration tab of a vSphere host, selecting Advanced Settings and selecting the node Disk. You should then browse to Disk.SchedNumReqOutstanding and set it to 256.
At the VM Virtual Hardware Level
It is critical that the the virtual hardware that is configured for a XenApp deployment be configured with the most efficient components as follows:
- Disable and remove any unnecessary virtual hardware components such as CD-ROM and Floppy
- When configuring a virtual network Interface Card, select the VMXNET3 adapter, this provides the most efficient vNIC possible and yields the best performance while minimizing load on the vSphere host CPU
- The ideal disk controller for XenApp is the LSI Logic SAS Controller. While converting a XenApp server from physical to VM is strongly discouraged (particularly because changing the disk controller results in an unstable operating system), if you have to convert, try and change the disk controller as this will affect performance.
- You can edit the VM settings and select the Options tab, configure CPU and meory hardware assist appropriately. By default it is set to Automatic; your testing might reveal a performance gain if hardcoded.
For 32-bit Windows Operating Systems
When using a 32-bit operating system, you are limited to the amount of memory that Windows 32-bit can effectively take advantage of. Therefore, I suggest CPU and memory configurations to be 4096 MB virtual RAM and one or two vCPUs. While you can most certainly configure four vCPUs, XenApp does not linearly scale very well. As such, two vCPUs is recommended before adding another XenApp VM.
Ideally, if you have to use 32-bit operating system, use Windows Server 2008, aside from the fact that Windows Server 2003 is almost end of life, Windows Server 2008 has many specific advantages, including one that automatically aligns the virtual disk to the underlining storage where it resides. This is significant from a disk I/O performance stand point.
However, if you must use Windows Server 2003, I suggest splitting the operating system and the data drive into separate VMDKs for the simple sake of being able to disk align the data drive. In Windows Server 2003, disk alignment is a manual process that should be performed on the data drive.
To align the drive on Windows Server 2003, use follow these steps:
- Add a Virtual Disk to the VM settings
- In the operating system, do not use the GUI to initialize the partition, instead open a command line and use the following command: Diskpart
- Then type list disk, this will display a list of available disks
- Use the select disk command in conjunction with the appropriate disk number to set the focus on the disk you want
- Finally type create partition primary align=64
- You can now use the Disk Management MMC to assign the disk a drive letter and format it
You will not be able to align the system drive and you don’t need to, considering that all your applications will be installed on the data drive anyway and that is the drive that will drive the disk I/O when the applications are used.
Multiple Drives for OS and Applications
I have read many times about the benefits of splitting the operating system and the applications onto separate drives. The fact of the matter is, if you are splitting them up logically but they physically reside on the same disk, then there is no performance benefit whatsoever. If you are splitting them up on different physical disks of different tier class (ex: SSD, SAS, SATA) with different RAID configurations, then a performance gain is noticeable, albeit not significant.
What I am trying to say here is, if you are going to have a dedicated VMDK for operating system and applications that live on the same datastore, then might as well have one large C: drive, as you will gain no performance at all. But if you plan on splitting your VMDKs on different datastores which have different tier of disk and RAID levels, then go for it.
General Recommendations
Disable the Last Time Access Attribute for NTFS process. It's a chatty process that keeps track of the last time a file was accessed. Disabling it relieves the operating system from having to constantly read and write this information. To disable, run the following command:
fsutil behavior set disablelastaccess 1
Finally, as with the build and setup of traditional XenApp servers, any unnecessary processes, services, USB, screen savers, animations, etc., should be disabled for best performance.
This process may seem long and exhausting at first, but isn’t that the case for every build of XenApp? Those of you that have been working with XenApp for a while can appreciate it this. That being said, virtualization makes it easier for us to create templates and deploy from these templates, thereby streamlining and efficiently increasing our ability to deploy a XenApp server.
Posted by Elias Khnaser on 03/14/20113 comments
What comes to mind as one of the biggest roadblocks to mass desktop virtualization adoption? Storage might come immediately to mind for most of you. We constantly hear about storage in terms of IOPS, cache and how much it will cost to implement desktop virtualization. Some have taken it even farther by suggesting local disk as an alternative, which gives better performance/lower cost at the expense of sacrificing virtualization features. Is there a way to have the best of both worlds?
More
Posted by Elias Khnaser on 03/08/20112 comments
The release of Citrix XenDesktop 5 introduced a powerful PowerShell engine that allows you to tweak and script almost every aspect of XenDesktop 5. This time, I want to show you a handy little tip that I frequently use when troubleshooting or even during a proof of concept.
When users connect to a XenDesktop 5-pooled VM, the default behavior is to restart that VM as soon as the user logs off. This action flushes all the changes that the user has made during the session and restores the VM to its pristine state, ready to accept a new user. This behavior, of course, only affects VMs that are based on a master image using MCS in Pooled mode or using Streamed with Provisioning Services.
While this behavior is very desirable when everything is working well and there are no issues, it can quickly become very annoying when you are troubleshooting, because the VM will reboot every time you log off. This is also an annoyance if you are in a POC and need to log off but don't want the VM to reboot.
To change this log-off behavior, you can use the PowerShell tab from the Citrix Desktop Studio console as follows:
- Load the correct Citrix modules that enable this feature. In most cases, these modules are enabled by default. But just in case they are not, run this command: Add-PSSnapin Citrix.*
- Identify which Desktop Group you want to apply this change against. You would typically do this against the Desktop group you are working on. To list the available Desktop Groups, run this: Get-BrokerDesktopGroup
- Once you have identified the Desktop Group, change the log-off behavior from "reboot VM" to "don't reboot VM" by running this command: Set-BrokerDesktopGroup -Name "DG Name" -ShutdownDesktopsAfterUse $False
When you are done troubleshooting or if for any reason you want to change the log-off behavior back to reboot VMs, run this command: Set-BrokerDesktopGroup -Name "DG Name" -ShutdownDesktopsAfterUse $True
PowerShell is very useful when working with software like XenDesktop, where you need to script changes.
Posted by Elias Khnaser on 03/02/20111 comments