Editor's Note: The head of customer care at Vision Solutions offers his account of events surrounding one of the most financially devastating North American weather-related events of 2011.

By the evening of Friday, August 26, the inhabitants of a long stretch of the eastern seaboard all had something in common: There was a likely chance there lives would be greatly impacted by a destructive hurricane. On that day, a record number of viewers were glued to the Weather Channel to watch the offspring of a distant tropical cyclone swell into a 600 mile-wide hurricane; with prolonged wind speeds in excess of 111 miles per hour.

And swell it did. In a short time, highly precise, satellite microwave-based instrumentation operated by NASA had recorded a rainfall total that raised eyebrows throughout the space agency. On the ground, farm fields, homes, warehouses, data centers—you name it—were swimming in the turbid floodwaters that Irene left behind.

As high winds and unrelenting rainfall pounded the coast and tornados formed to the south, Pete Robie watched intently from 2,800 miles away in Irvine, California. As Vision Solutions’ executive vice president of customer care, ultimate responsibility for the welfare of customers who stood in Irene’s path, fell to Robie. Banks, hospitals, hotels, casinos, manufacturers and other businesses that contracted with the HA/DR software provider, were all—in the end—counting on one thing.

The phone began to ring on Friday evening. While IT professionals are typically thought of as logical and not easily rattled, according to Robie, an even-keeled person with over 27 years of experience in technical support, “You could hear these things unfolding in the background and you could sense how unsettled the person on the other end of the phone was. You could tell they were under intense pressure and the only thing that was important at that moment, after the health and welfare of their people, was protecting the assets for which they were responsible.” 

A Confluence of Resources

Robie began working for Vision Solutions in 2001, long before Thoma Bravo, LLC acquired that company and subsequently iTera, Lakeview Technology and Double-Take Software. In short order, Robie was responsible for the care of over 25,000 customers all around the world. He reshaped his far-flung technical support team by providing cross-training, a vast knowledge base, and new technology. The innovative technology would facilitate a global view of open- and closed-support tickets, as well as foster better communications between the groups. He also shepherded Vision’s technical support department through the Service Capability and Performance (SCP) Standards certification process. SCP certification is the global benchmark for service excellence.

“High availability systems are as critical to the health of a business as core applications. Having a knowledgeable and empathetic technical support person available to you is as important as having HA software.  That’s why we’ve invested so much effort in creating a support environment that exceeds the standard for acceptable service.”

Since Vision has customers worldwide, Robie must track acts of God and acts of man in every time zone. “We have large flat screens here in Irvine displaying global and local weather patterns, and we receive alerts from several agencies on potential threats. That way, we can prepare for these events. We can also increase staff levels to accommodate an elevated number of calls,” says Robie.   

Vision uses a CRM system, modified with event flags that are attached to customer records. If a customer—say—is in the path of a hurricane, a member of Robie’s support team can flag that record. Before the storm hits, a support representative calls and emails that customer. “We try to give them advance notice and let them know we’re here to help. It’s easier to help them prepare for a disaster by failing over to their secondary server and powering down their primary than it is to put all of the pieces back together in the aftermath,” says Robie. So how does he know that most customers do their own failover well in advance of a disaster? Customers share this information in the proactive phone calls.

Continuity of care is as important when providing technical support, as it is when practicing emergency medicine.  At Vision, every support ticket is assigned to specific technician. Each technician follows the event from when it is first reported through to its resolution.

Despite the anxiety that precedes an event like this, most people will not board up the windows, stack sandbags by the door, or cut over to their backup server. “You can’t tell exactly how a storm is going to track, or how it will affect people on the fringes,” says Robie. “It could drift 50 miles north or south of the target,” he says. In instances like these, you don’t really know how bad things are going to get until the last minute, and things can get very tricky.”

Irene, and numerous tornados that hit the area, left over two million utility customers without power. As electrical service is restored, Vision’s customers would need help switching operations back to their primary production servers. “Ironically, there is a lot of work to do long after the hurricane winds down,” says Robie. “At first, when they switch over to the backup server, there’s a sense of relief and users are unaware of the switch.”  He adds that it can be difficult to return service to the primary system since, in some cases, the equipment is totally destroyed or needs to be rebuilt.  Yet, it is essential that Vision’s customers return their production computing workloads to their primary server so they are ready for the next hurricane, tornado, or general hardware failure.

Not Everyone Calls

According to Robie, Vision has a large number of customers who do periodic failover tests, and they know from experience how things are going to work. “Most of our Mimix, iTera and Double-Take users (who were in the path of this hurricane) actually failed over to a backup server outside the affected area before the hurricane came to shore.

”Yet, he says, a small percentage of people who don’t failover in advance or don’t need guidance with Vision’s HA solutions, will call the company’s customer care line. As Irene wrought mayhem on the Atlantic seaboard, there were only four documented calls placed to technical support directly related to the storm. Vision’s technicians stepped in and helped these clients move their workloads to backup servers on higher ground. “Once the decision is made, the switch can happen in two to twenty minutes depending on how much preparation needs to be done.  If they have a large number of connections to peripherals, then it takes longer. Afterward, we have to verify that everything is working,” says Robie. Vision’s support technicians have the ability to assume control of the customer’s console after consent has been given. “We can do 99 percent of everything we need to do remotely. We do not make any changes to the customer's system without their permission and the customer is always aware of what we are doing.”

In one of these instances, he says, a bank with 24/7 operations called and requested assistance before they got hit by the brunt of the storm. “They were a new customer and didn’t have much experience failing over to the backup server. Every hour or two, we called them, or they called us. One member of my staff worked with this customer from Friday though Sunday to make sure their systems and data were protected. In the end, everything worked perfectly.”

Irene is now part of the past. Starting out as a Category 3 hurricane, it dissipated into a very large Category 1 storm after hitting the east coast of the U.S. It caused 55 deaths and more than $10 billion in property damage in all. Its impact on Vision’s customers has been documented and several letters have been received from customers expressing their appreciation. One such letter from an IT director at an insurance company in Rhode Island explains how one of Vision’s technicians essentially saved the company's entire email environment. The author adds that wherever his career takes him, Vision’s HA technology will be at the top of his recommendation list.

Tomorrow, new hurricanes, cyclones, tornadoes will form throughout the world and Robie will track them and assess their capacity for destruction.  Other critical situations will arise—some natural and others not. In all cases, customers in need of urgent attention will have an ally available to help address their HA and DR challenges.