23
Sat, Nov
1 New Articles

Anticipating Hurricane Irene

Channel News
Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

Editor's Note: The head of customer care at Vision Solutions offers his account of events surrounding one of the most financially devastating North American weather-related events of 2011.

By the evening of Friday, August 26, the inhabitants of a long stretch of the eastern seaboard all had something in common: There was a likely chance there lives would be greatly impacted by a destructive hurricane. On that day, a record number of viewers were glued to the Weather Channel to watch the offspring of a distant tropical cyclone swell into a 600 mile-wide hurricane; with prolonged wind speeds in excess of 111 miles per hour.

And swell it did. In a short time, highly precise, satellite microwave-based instrumentation operated by NASA had recorded a rainfall total that raised eyebrows throughout the space agency. On the ground, farm fields, homes, warehouses, data centers—you name it—were swimming in the turbid floodwaters that Irene left behind.

As high winds and unrelenting rainfall pounded the coast and tornados formed to the south, Pete Robie watched intently from 2,800 miles away in Irvine, California. As Vision Solutions’ executive vice president of customer care, ultimate responsibility for the welfare of customers who stood in Irene’s path, fell to Robie. Banks, hospitals, hotels, casinos, manufacturers and other businesses that contracted with the HA/DR software provider, were all—in the end—counting on one thing.

The phone began to ring on Friday evening. While IT professionals are typically thought of as logical and not easily rattled, according to Robie, an even-keeled person with over 27 years of experience in technical support, “You could hear these things unfolding in the background and you could sense how unsettled the person on the other end of the phone was. You could tell they were under intense pressure and the only thing that was important at that moment, after the health and welfare of their people, was protecting the assets for which they were responsible.” 

A Confluence of Resources

Robie began working for Vision Solutions in 2001, long before Thoma Bravo, LLC acquired that company and subsequently iTera, Lakeview Technology and Double-Take Software. In short order, Robie was responsible for the care of over 25,000 customers all around the world. He reshaped his far-flung technical support team by providing cross-training, a vast knowledge base, and new technology. The innovative technology would facilitate a global view of open- and closed-support tickets, as well as foster better communications between the groups. He also shepherded Vision’s technical support department through the Service Capability and Performance (SCP) Standards certification process. SCP certification is the global benchmark for service excellence.

“High availability systems are as critical to the health of a business as core applications. Having a knowledgeable and empathetic technical support person available to you is as important as having HA software.  That’s why we’ve invested so much effort in creating a support environment that exceeds the standard for acceptable service.”

Since Vision has customers worldwide, Robie must track acts of God and acts of man in every time zone. “We have large flat screens here in Irvine displaying global and local weather patterns, and we receive alerts from several agencies on potential threats. That way, we can prepare for these events. We can also increase staff levels to accommodate an elevated number of calls,” says Robie.   

Vision uses a CRM system, modified with event flags that are attached to customer records. If a customer—say—is in the path of a hurricane, a member of Robie’s support team can flag that record. Before the storm hits, a support representative calls and emails that customer. “We try to give them advance notice and let them know we’re here to help. It’s easier to help them prepare for a disaster by failing over to their secondary server and powering down their primary than it is to put all of the pieces back together in the aftermath,” says Robie. So how does he know that most customers do their own failover well in advance of a disaster? Customers share this information in the proactive phone calls.

Continuity of care is as important when providing technical support, as it is when practicing emergency medicine.  At Vision, every support ticket is assigned to specific technician. Each technician follows the event from when it is first reported through to its resolution.

Despite the anxiety that precedes an event like this, most people will not board up the windows, stack sandbags by the door, or cut over to their backup server. “You can’t tell exactly how a storm is going to track, or how it will affect people on the fringes,” says Robie. “It could drift 50 miles north or south of the target,” he says. In instances like these, you don’t really know how bad things are going to get until the last minute, and things can get very tricky.”

Irene, and numerous tornados that hit the area, left over two million utility customers without power. As electrical service is restored, Vision’s customers would need help switching operations back to their primary production servers. “Ironically, there is a lot of work to do long after the hurricane winds down,” says Robie. “At first, when they switch over to the backup server, there’s a sense of relief and users are unaware of the switch.”  He adds that it can be difficult to return service to the primary system since, in some cases, the equipment is totally destroyed or needs to be rebuilt.  Yet, it is essential that Vision’s customers return their production computing workloads to their primary server so they are ready for the next hurricane, tornado, or general hardware failure.

Not Everyone Calls

According to Robie, Vision has a large number of customers who do periodic failover tests, and they know from experience how things are going to work. “Most of our Mimix, iTera and Double-Take users (who were in the path of this hurricane) actually failed over to a backup server outside the affected area before the hurricane came to shore.

”Yet, he says, a small percentage of people who don’t failover in advance or don’t need guidance with Vision’s HA solutions, will call the company’s customer care line. As Irene wrought mayhem on the Atlantic seaboard, there were only four documented calls placed to technical support directly related to the storm. Vision’s technicians stepped in and helped these clients move their workloads to backup servers on higher ground. “Once the decision is made, the switch can happen in two to twenty minutes depending on how much preparation needs to be done.  If they have a large number of connections to peripherals, then it takes longer. Afterward, we have to verify that everything is working,” says Robie. Vision’s support technicians have the ability to assume control of the customer’s console after consent has been given. “We can do 99 percent of everything we need to do remotely. We do not make any changes to the customer's system without their permission and the customer is always aware of what we are doing.”

In one of these instances, he says, a bank with 24/7 operations called and requested assistance before they got hit by the brunt of the storm. “They were a new customer and didn’t have much experience failing over to the backup server. Every hour or two, we called them, or they called us. One member of my staff worked with this customer from Friday though Sunday to make sure their systems and data were protected. In the end, everything worked perfectly.”

Irene is now part of the past. Starting out as a Category 3 hurricane, it dissipated into a very large Category 1 storm after hitting the east coast of the U.S. It caused 55 deaths and more than $10 billion in property damage in all. Its impact on Vision’s customers has been documented and several letters have been received from customers expressing their appreciation. One such letter from an IT director at an insurance company in Rhode Island explains how one of Vision’s technicians essentially saved the company's entire email environment. The author adds that wherever his career takes him, Vision’s HA technology will be at the top of his recommendation list.

Tomorrow, new hurricanes, cyclones, tornadoes will form throughout the world and Robie will track them and assess their capacity for destruction.  Other critical situations will arise—some natural and others not. In all cases, customers in need of urgent attention will have an ally available to help address their HA and DR challenges.

MC Press Online's staff of reporters and editors are trained professionals who share a background in journalism and have a passion for technology. Many have been in the IBM midrange industry for more than 20 years and have a keen sense of what's important to IBM i and Power Systems users. If you have a news tip, story idea, or press release, email it to This email address is being protected from spambots. You need JavaScript enabled to view it..

BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: