Home > Articles By Issue > Safety & Security > Article Aug. 2002

Disaster Recovery And Contingency Planning In Extreme Times

As part of the magazine's ongoing coverage of The TFM Show, this column showcases one of 2002's informative pre-conference roundtables. It includes questions asked by moderator Heidi Schwartz and answers provided by the roundtable of experts (listed below).

In 2003, roundtables will be a key part of The TFM Show program, so be sure to register early. Visit the Web at www.tfmshow if you'd like to attend next year's show.

Disaster Planning Experts:
George Fenton, BELFOR/INRECON
David McDaniel, BMS Catastrophe Special Technologies Division
Dallas Dean, American Power Conversion

Schwartz: What is the biggest change in disaster planning since 9/11?

Fenton: I'd have to say it's the realization that the same kind of thing could happen to anyone—even if it were directed at someone else. You could own a business in Southern California or anywhere, and an attack could happen to someone just down the road.

Disaster planning is more important now than ever, but I could have said that a year ago. This one incident (9/11) really lit it up for everybody. Yet there are several questions that still remain. What does disaster planning do? How do you prepare for it? What does it mean for your organization?

McDaniel: I think we learned from Oklahoma City that anybody can get hit. The one thing that people realized on 9/11 was the fact that they didn't have anything to go back to. The papers were in the street, and the data was destroyed. Remote data saving is an approach that identifies the vital five to seven total documents that your business depends on and finds a way to protect them.

Those people in information technology have been thrown into a tizzy, because now the server can't be in the same facility. The only problem is it's slow, it's file oriented, and it can only move out so far.

So how do you preserve the data, and how do you have backup in several locations? Do you continue business today realizing there maybe nothing of the original infrastructure that you can go back to?

Dean: In terms of electrical infrastructure, there has not been a whole lot of additional change because of the events of 9/11. However, I will add that no amount of planning could help under those circumstances, but now people are thinking about backup facilities and other options when something catastrophic happens to their primary facility.

Schwartz: How many people made serious changes to there disaster plans because of 9/11?

Fenton: You know, disaster planing grew out of information technology systems (ITS), so it really hasn't spread so rapidly into facilities. Up until now, it has been about books, information, documents, records, and those types of things. But now, these are the things that people are starting to think about—more so than just the focus on the computers. There are many pieces to it.

McDaniel: I have talked to several people, and they tell me they became very interested in disaster planning after 9/11. What typically happens is they make some changes, but it doesn't take long for the thing to kind of drift, and nothing really sticks; all the interest goes away.

But if facility managers consider things in terms of business impact analysis, they'll understand how important this actually is these days. This approach examines how many dollars per day are lost if the company is not in business.

Those professionals in the financial services field understand the importance of this step, because the government has stepped in and said, "Okay, banks, you have to have a disaster recovery plan. But now the facility people in production and manufacturing are suddenly noticing they may have the same problem.

Business impact analysis examines the hierarchy within the department. It identifies all that is critical—not only documents, equipment, and infrastructure. It looks at where the money comes from and where it goes. It also answers the question, "How long can you be out of business before you're out of business."

Schwartz: Could you please identify the top threats for facility managers? Why are these so devastating?

Dean: Number one is utility failure—especially in California. In the last year or so, it has been touch and go with the major utility companies going bankrupt and everyone experiencing power shortages. This has been on everybody's mind, and it has been the source of many upgraded critical power systems. Now it has died down a little, but for a while it was priority one. Just last summer, the power companies were warning of one or two blackouts city wide every week.

McDaniel: I agree with regard to utility failure, but water inundation is also very important. It can come from an overflowing toilet, a burst pipe overhead, floods, hurricanes, or many other unpredictable sources. Then there's always the follow up to avoid secondary damage.

Fenton: Water is very serious indeed, but you can't cookie cutter everything. What is important for one business isn't necessarily crucial for another. For instance, what's crucial for a major retail office may not impact a high rise building.

Facility professionals must ask themselves what their business is really about. How do I keep my customers from leaving? If I can be back up in business very quickly, that helps not just me and my employees, but it also helps me to serve my customers well.

Several years ago, I worked on a large job for a major retail company. When the company was hit by a tornado, some customers were told that the store was closed for 10 days for repairs. Consequently, part of the disaster recovery plan was to bus the customers to another store—rather than allow them to go across the street to the competitor.

It's very important—for a number of reasons—to assess the business and understand customers as well as processes. Of course there are the critical tangibles—books, technology, computers, and everything else that makes a business go—but facility professionals have to think outside the box a little bit.

Schwartz: What is the most common mistake made in terms of disaster planning?

McDaniel: One areas that's most often overlooked is the people who put together the disaster plan and try to keep the business going. Facilities managers deal with the original location—the nitty gritty of putting it back together and getting ready to get back to business in a normal fashion. Disaster recovery planners sit in their ivory towers; they don't really know what they are dealing with.

Fenton: I think another big problem is ownership of the disaster recovery plan. Everyone is looking for an answer, and they think there's a document somewhere out there that will answer all their problems. Whether a risk manager, facility manager, or information technology manager gets involved, the plan really needs to be a company owned deal. It does no good for everyone to fight for turf.

It is very important to get a known point person who can motivate everyone to work together. Once the plan is completed, it can be put in a file and examined on a regular basis. And if there's an emergency, they can bring it out and figure out exactly what does or doesn't work.

A disaster recovery plan is a living, breathing document based on a revolving process. If something doesn't make sense or is overly complicated, then reconsider your options.

All managers need to be on the same page in terms of what they want to accomplish. Unfortunately, disaster planning is often a reactionary process. Managers and insurance people don't want to deal with it ahead of time.

Clearly, a company is much better off with a good plan in place ahead of time. Imagine what would happen if the key players in a company were forced to come up with something in the throes of utter chaos!

Schwartz: More specifically, what kind of role does the relationship between IT and facilities have in the development of a disaster recovery plan? How can it be coordinated in a way that produces an overall benefit to the company?

Dean: Most of the time, it's contentious, to say the least. IT has its goals (like zero down time, which might come from the facility side of things). IT doesn't pay much attention to what facilities people are saying. In their defense, IT people often don't understand how much work it takes, because whenever they get a request, the turnaround time is very short. In terms of disaster recovery, I think it helps to have both sides sit down together and work out a plan.

We just recently went through a year long interview process with Fortune 500 companies. Both IT and facilities people came together to develop some kind of system that would help them work things out in the beginning. This would help them understand how to grow their electrical infrastructure and avoid a last minute, frantic rush.

Just sitting down together is hard, but that is really the key—especially lately. Unfortunately, with many of the most recent disasters, the left hand didn't know what the right hand was doing.

Schwartz: How do insurance companies help facilities managers with disaster recovery planning?

Fenton: I think it's changing very, very slowly. Insurance companies are very loss oriented, which means they take zero action before a claim takes place. However, some companies do give great deals for safety oriented organizations.

The guys that are selling the service are better prepared in the event of a loss. They keep the loss ratio down, and they get the biggest charge for everything else. It's not so much the insurance but all the people who are taking a larger percentage of the pie.

McDaniel: Many major insurance companies will help you with risk engineering management. You can call your broker and get someone to come in, look at your plant, and make some suggestions.

The whole process in disaster recovery planning is going in and recognizing the things that threaten to stop your business. Then you make the decision whether to be proactive or reactive.