Rare, twin natural disasters hit the densely populated northeast portion of the United States this past week. First, an earthquake centered in Virginia caused minor damage, with vibrations felt from the Carolinas to Canada. Then, Hurricane Irene swept up the East Coast, killing 10 and causing what will certainly be billions in property damage. Both events were a stark reminder of the unpredictability of nature and the importance of sound business continuity planning.
For decades large enterprises have had formal programs in place to ensure that a major event (weather, flooding, fire) impacting a primary data center would not seriously hurt their ability to service their customers. Typically, this was achieved by creating a backup data center far enough from the primary to ensure that any single event would not impact both locations. Unfortunately, some major trends have made simple data center recovery only a starting point for business continuity.
The ascendency of mission critical, customer facing applications is a huge driver of business continuity requirements. Most firms have a significant base of these applications, with strong customer expectations of round-the-clock availability. In addition to increased customer expectations, the threat landscape has grown more complicated and unpredictable. Traditionally, planning focused on “conventional” data center disruptions noted above. In the past 10 years we have seen terrorist events, pandemics and civil disorder added to the list of planning assumptions. With most employees being technology dependent to service customers, office locations become critical resources just like data centers. Finally, the increase in outsourced services (cloud based or traditional), raises the complexity of ensuring robust recovery capability.
In order to ensure a strong business continuity capability, forward thinking firms would be wise to consider the following best practices:
Data Center Backup
At a minimum, firms should have a secondary data center, geographically remote from their primary. A traditional rule of thumb has been 200 miles of separation. However, recent changes in the threat landscape have caused many firms to rethink this assumption. Many firms are now moving to bicoastal data centers or separations in the 1000+ mile range. This level of separation has typically been difficult due to bandwidth, communication latency and cost issues. In order to replicate large amounts of real time data quickly and cost effectively, data centers had to be located within a short (typically metropolitan area) distance.
For a number of firms this geographic challenge still exists. Therefore, the new gold standard for recovery involves 3 environments. Two of these environments are “high-availability” (or HA) with real-time failover capability. These environments are typically located within the same physical data center or within the same metropolitan area. A third environment, serving as a disaster recovery center is located in a remote geographical area.
As mentioned above, the increasing trend towards using service providers has complicated the recovery process. It is important to understand your provider’s architecture and recovery capabilities. This is complicated by the fact that many service provider environments utilize virtualized and shared infrastructures. At the same time, a well implemented (and managed) service provider environment can be a plus for a firm’s business continuity program. It can allow for further diversity of resources, making it less likely that a single event will create a large recovery requirement.
Work From Home
Having a strong work from home program is the closest thing to a silver bullet in the business continuity world. When employees can successfully perform their job function from home (or any remote location), they are able to recover from a vast array of threats and incidents. It provides tremendous flexibility when dealing with situations in which the workplace is no longer a suitable location. In order to make work from home programs successful, the following capabilities need to be implemented:
- A strong, secure remote access capability must be in place.
- Desktop VOIP and chat capabilities must be available
- Their must be no dependence on localized desktop resources. That is, no reliance on a remote desktop for applications or data.
- Elimination of manual, paper based functions. File cabinets, forms, and ink are the bane of recoverable processes.
- A strong knowledge management capability must be in place that allows for the location of procedures, processes, documentation and online forms.
- Regular use of the facility in daily operation to test its availability
While a recovered data center and a work from home program are an excellent foundation for a business continuity program, they can’t ensure that an office or region won’t be impacted to the point of paralysis. That would prevent even home based workers from servicing customers. An additional critical element of a business continuity program is a “buddy system”. For every critical employee, a buddy is assigned in a geographically remote location. That buddy has all of the knowledge and tools to perform the critical function. The gold standard would have a third person as an additional backup. The buddy system is predicated on the idea that a strong data center recovery program and work from home capability are already in place.
This blog post is simply meant to be a conversation starter regarding business continuity best practices. If your firm has not already considered these points, with some material efforts in place, you should be concerned and moving forward with planning. If you have considered these practices, hopefully you have implemented them in a comprehensive and professional manner. Depending on the size of your enterprise, you should have dedicated, internal business continuity professionals and/or experienced consulting assistance. Like insurance, business continuity can be an unpleasant and seemingly low priority endeavor, unneeded for daily successful business operation. Unfortunately, as with insurance, by the time you realize you need it, it’s too late, with devastating consequences.