4 Ways to Make Disaster Recovery Testing Less Painful

In the film, Peter Sellers plays three roles--all to perfection--but it is as the demented former Nazi and weapons expert, Dr. Strangelove, that he says to the Soviet ambassador, Alexei de Sadiski, "The whole point of the Doomsday Machine is lost if you keep it a secret."

Forty-seven years later, the whole point of having DR systems may also be lost if they are not tested, yet in many cases, that is the case. Jon Toigo, CEO of IT consultancy Toigo Partners International, warns against such omissions, saying, "Testing is the long tail cost of DR planning and not thinking about testing up front, when selecting strategies, will come back to bite you in the butt down the line."

So why the laissez-faire attitude? Here are four primary reasons, and suggested antidotes.

Reason 1: It's a hassle. Many companies don't test their DR systems because it's just too cumbersome in complex IT environments where systems must be up and running 24/7, notes Bob Roudebush, VP of marketing at Neverfail Group. "Of course, the negative ramifications are huge," he adds. "If you don't test your DR, it might not work in a disaster, or you might not know the right steps to take to get your systems back up and running."

Antidote: Roudebush says companies should look to implement solutions that have easy testing capabilities built in. These solutions should minimize the time it takes to test systems, should provide automatic testing functionality and should not cause disruption to running applications and systems.

"IT teams should try to test systems at least twice a year to ensure staff knows how to deal with a disaster, he says. "As is the case with smoke alarms, test when the clocks change for Daylight Savings Time and Standard Time, so it's easy to remember."

Reason 2: Testing is misunderstood. According to Toigo, "People are afraid that they just spent a boatload of money on some data protection scheme, and if they test it, they may fail."

Antidote: In Toigo's opinion, a primary purpose of testing is generating failures so DR systems can be improved. Adds William Hughes, Director, Consulting services BC/DR Center of Excellence at SunGard Availability Services, organizations need to promote the view that a test doesn't need to be passed; it needs to promote improvement. Toigo sums up by declaring: "There are no failed tests."

Reason 3: Resource Restrictions. IT budgets remain tight, and doing more with less is still frequently the corporate mantra. As a result, CIOs or senior IT execs who signed off on purchasing data protection products one day may turn around and say there is no more money to test them the next. "Projects are generally reducing costs or creating opportunities," says Hughes. "It's very hard to impact either one of those areas."

Antidote: Hughes says organizations should re-evaluate their validation and testing processes, and determine ways to minimize the impact on them. That could involve using a third party to conduct the exercise or simply using less constrained resources to participate.

Reason 4: What's it Worth to Me? The lack of perceived value is a major DR testing bugaboo that results when DR testing is viewed as insurance that may never show any kind of ROI.

Antitodote: According to Hughes, "This is an issue that plays with resource constraints, since it's certainly harder to extract resources to support something that's not perceived as having high value." He goes on to note that this can happen because the organization looks at the implementation of the solution as the end point, rather than an inflection point. Or it may be that the organization has validated the solution a number of times and sees it as routine or easy. By using testing to continuously raise expectations, there is an ongoing state of improvement that reflects the increased value of testing.

Concludes Toigo: "A disaster is a messy thing by nature, and you can't anticipate all the messiness, so you do your best to test, test, test, because that is essentially a rehearsal. William Shakespeare said "All things are ready if our minds be so."

Posted by Bruce Hoard on 03/16/2011 at 12:48 PM