Dan's Take

Does the Survey Really Say What the Survey Says?

Deciphering a CloudPhysics survey on application disruption.

• By Dan Kusnetzky
• 10/23/2015

CloudPhysics sent me a press release touting the results of an "Application Disruption" survey the company conducted at VMware's VMworld U.S. event. More than 1,000 of the 23,000 event attendees participated.

John Blumenthal, co-founder and VP of Strategy at CloudPhysics, pointed out that "The findings clearly highlight the complexity VMware admins deal with on a daily basis." He went on to say "Complexity prevents them from seeing the consequences of configuration changes: both immediate impacts on applications and the creation of hidden risks that degrade the infrastructure over time. Managing change in a complex environment is beyond unaided human judgment; analytics must accompany and drive optimizations to protect against disruptions to business operations."

Survey Highlights
Here are the results CloudPhysics highlighted in the release:

• 68% of respondents indicated that the top cause of disruptions are configuration changes, highlighting the complexity facing VMware admins
• Adding to the challenge, more than 50% are trying to manage their virtual environments with VMware vCenter alone, unaided by dedicated operational management tools
• While admins recognize the value in additional management tools, more than 65% reported taking more than a month to implement an on-premise management solution, and of those, more than 45% reported it taking more than three months
• Operational challenges vary by segment: while the top initiative for large enterprises is creating a software-defined datacenter (SDDC), commercial organizations and SMB companies are focused on hardware and OS upgrades -- 44% and 48% of respondents, respectfully

One of the challenges facing the consumer of survey-based data is determining if the results are meaningful, useful or relevant to the decisions the consumer faces. Vendor-sponsored surveys can be self-serving, limited and possibly misleading. Even though the survey's response could be seen as limited and not truly representative of the market as a whole, the sponsor often touts the results as being relevant to the entire market.

CloudPhysics, on the other hand, was careful to present that the survey was conducted at a vendor-sponsored event and how many of the 23,000 event attendees participated. The company also pointed out that the responses were related to VMware's virtual machine (VM) software, its management tools and some of its marketing initiatives.

Dan's Take: Data Analysis As It Relates to Surveys
A broader question, however, is whether 1,000 attendees of a vendor-sponsored event, even an event such as VMworld, represent the thinking and opinions of the entire worldwide market for VM technology, or those wanting to move to an SDDC? Do users of Xen, regardless of whether it's from Citrix or Oracle, or KVM, regardless of whether it's from IBM or Red Hat (or some other Linux distributor) face the same challenges? Would these organizations offer similar responses to the survey questions? This, of course, is not at all clear because the survey was limited.

In the end, if your organization uses VMware and its products, the results could be both useful and relevant. It also would tend to support CloudPhysics' message that it would be wise for organizations in this category to consider tools that can continually gather VM operational data, analyze it and offer actionable suggestions about improving the environment.

If your organization uses a mix of virtualization technologies coming from a number of suppliers, or just uses virtualization technology from one supplier, and it isn't VMware, the results, while interesting, are largely irrelevant.

I applaud CloudPhysics for its transparency on where the survey was conducted, a description of the respondents and where the results would and cloud be useful. The company didn't fall into the trap of making broad claims based upon a limited study.