Fri, Jul
4 New Articles

The DR Plan That Went Wrong

  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

The trick with any recovery is to expect the unexpected.

By Steve Pitcher

It was Friday morning at ABC Widget Manufacturing Company, and things were business as usual. The machines were humming, and the accountants were accounting. Everyone was typing away at their green-screen terminals with thoughts of a wonderful weekend ahead.

That was until someone noticed water seeping out from under the server room door. Water was to be expected only if there was a fire in the computer room. The most flammable things in there were the stacks and stacks of green bar paper that had been collecting since 1978. You see, at ABC, they pride themselves on always being able to locate a paper copy of a transaction. If they bought a part for their widget machine in 1978 and it broke, someone would be able to track down the original purchase order and order that same part again. A handy system.

The old green bar paper, arguably, was what absorbed most of the cooling from the 15-year-old water-cooled air conditioning unit. Since the little Power Systems 720 sitting in the back ran the whole business, there was plenty of room to stack so much paper. Owen (the guy who changed the LTO backup tapes) passed by the 720 every time he went through the computer room's unsecured back door to have a smoke three or four times a day. One of those times, a little piece of ash flicked on the green bar and ended up starting a fire.

Unfortunately, there were no smoke-activated fire alarms in the computer room. There were only old-school fire sprinklers. So when the noisy little 720 stopped, you'd have thought someone would have noticed the eerie silence.

Fast-forward to a week later. The server room has been dried out, and the 720 has been replaced with another little 720 from the used marketplace plus a brand-new second-hand external 3580 tape drive. Luckily, ABC was diligent in doing regular full-system saves every three years (sarcasm intended). The Go Save 21 was their ace in the hole for a disaster recovery. That's all you need, right?

Unfortunately, they had no LIC DVD to boot the new system.

Fast-forward another 48 hours. The LIC DVD has been found for IBM i 7.1 in Owen's basement, where he keeps the backup tapes.

Fast-forward yet another 24 hours. Bare-metal restores take a whole lot of time when you don't do them ever. Lots of manual research and researching manuals. Plus calls to anyone in the address book who's ever done a bare-metal restore.

Fast-forward 24 more hours. Ready to load the full-system-save tape. More manual research and researching manuals. Finally, the system is back up to where it was a little under three years ago. Too bad the LIC and the OS were at different PTF levels. The LIC was 7.1 vanilla, yet the OS was at Technology Refresh 8. Looks like they're going to have to burn a Technology Refresh 8 resave of the I-BASE DVD and slip the LIC. Someone has to figure out the ESS website and get past the entitlement issues with the new serial number. They could've gotten it from the old serial number entitlement, but nobody's ever logged into ESS and set themselves up.

Fast-forward 72 more hours. Owen is asleep and not answering his phone. Recovery is tiring. He's had a tough week. Too bad the backup tapes are in his basement. They'll have to wait until he comes in Monday.

Fast-forward to Monday. Time to restore from the nightly tape! Hang on! Every night they backed up only saved changed objects. They need Owen to go get all the tapes and explain what the heck tape goes next.

About this time, the salvaged remnants of the green bar paper are almost dry enough to flip through.

Of course, this is an extreme example of a horrible situation brought on by carelessness, unpreparedness, and just plain bad luck. The trick with any recovery is to expect the unexpected. Everything will go wrong at least once, and you hope someone learns from it.

I'm no expert in disaster recovery, but I've seen my share of things that have gone wrong. I've witnessed everything from exploding propane generators to under-powered uninterruptible power supplies dropping systems hard. I've watched seasoned professionals accidentally pull live disk drives from running systems while replacing failed drives, and I've seen fibre lines cut with circular saws. I've seen core network switches plugged into outlets that are tied into light switches that stop working when the server room lights get turned off. I've seen tarps and umbrellas and shingles and gutters inside server rooms to displace running water.


I've seen gutters.

Gutters installed on racks.

Anything can and will happen because it repeatedly does, even to the best of them.

What doesn't happen repeatedly enough are controlled disaster recovery tests. Some companies do it. Not enough do. But those who do it regularly have it down to a science. It's part of the overall strategy. Mandated even so that everyone knows their role and is held accountable for the success of it.

One question I hear when talking about disaster recovery is always: How fast can you recover? It's simple. Assuming you have a disaster recovery plan, those who test often will recover much faster than those who seldom test or never test at all. You test until you get to a point where you know how long to expect in a failover if there are no problems. That's your normal. It could be two hours, or it could be two days, depending on the business and type of disaster recovery system used.

With security being such a hot-button item nowadays, the breaches and fallout we tend to read about are not just security failures but also recovery failures. A major security failure can be partially remedied by a solid, tested recovery plan (no recovery plan can resolve the legal issues). For instance, if you had a server that was the target of some malicious code that deleted or encrypted hundreds of thousands of files, you'll usually have a few questions:

  1. Can we recover?
  2. If yes, how fast can we recover?
  3. How can we prevent this from happening again once we've recovered?

In this situation, arguably question 3 needs to be figured out before the recovery effort starts. More unforeseen downtime.

I was dealing with a similar situation a few years ago: A user PC had encrypted the file shares on a critical Windows 2012 server. Hundreds of thousands of files were unusable. Recovering the system wasn't a problem. Locating and stopping the user's PC from encrypting those files again was a bigger priority. Determining it was a user's PC causing the problem took time as well. If I recall correctly, the better part of 10 hours was spent with all IT hands on deck before the infected PC was located and disconnected and the file recovery completed to 24 hours prior. All user work done that day was lost. That's a situation where the backup was readily available and tested and the right people were on hand to work. All it takes is an additional variable to throw a monkey wrench into a recovery plan.

Testing a static plan isn't enough either. We need to plan for the worst-case scenario by making recovery dependencies unavailable during a test. What if the tape drive in your cold-DR location isn't functional? How will you replace it? Remove it from the equation, and see how your team reacts to the problem.

Expect the unexpected. Test for the unexpected. Because disastrous situations seldom cooperate with a plan.

Steve Pitcher
Steve Pitcher works with iTech Solutions, an IBM Premier Business Partner. He is a specialist in IBM i and IBM Power Systems solutions since 2001. Feel free to contact him directly This email address is being protected from spambots. You need JavaScript enabled to view it..


Support MC Press Online

$0.00 Raised:

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: