12
Sat, Oct
6 New Articles

Documenting the High Availability Environment

High Availability / Disaster Recovery
Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

By deploying high availability (HA) technology, you can shrink disaster-induced system downtime to 20 minutes or less. You can also deliver uninterrupted system access while maintenance activities are being performed.

HA systems can sometimes appear inordinately abstract, though, especially when the chips are down and failover procedures are not working as they should. Printed HA documentation can take the mystery out of a system failure and is indispensable to anyone who wants to maintain control.

Good HA documentation has two parts. Both parts share the same objective, which is to help administrators and others avoid getting caught in the slipstream of confusion and quickly restore access to mission-critical business functionality.

The first part of an HA document clarifies procedural uncertainties and sheds light on HA priorities, government regulations, and recent changes to system settings, among other things that, if not duly noted, will only prolong the misery. And to ensure its accuracy and viability, it must undergo periodic review.

The second component is a log that details such things as occurrences of exceptional system behavior and the actions taken to correct these aberrations. Even the most automated and technologically advanced HA system relies on the success of its smallest components.

Document Number One: The HA Operations Piece

There are many ways to structure an HA operations document, but it should always contain the following elements.

Business Requirements

Include a couple of paragraphs on why your organization has HA in the first place. Are you using it to create a true HA environment or just a backup tool? This will help technicians in charge of recovery understand what is a priority and what is not. Most often, companies implement HA to stem financial losses associated with system inaccessibility, but the central mission of HA might more directly be to improve business continuity, ensure system availability, or protect data.

Performance Benchmarks

Your best shot at a speedy recovery is to have a backup system that has the same processor functionality, memory, and disk sizes as the primary system. Although HA systems can function competently in situations in which the primary and backup systems differ in performance, matching operating system release levels and application software programs can support a more complete recovery.

Also, performance of the entire HA system depends upon the communications link between the two systems. If you don't have good communications, you will not have high-performance HA.

That said, many organizations save money by deploying backup systems that have less horsepower. Sometimes, they'll use an old production system as their backup, and on other occasions, they'll acquire a system that has been specifically sized to handle a subset of the primary system's functionality. Most HA experts don't like this option and say it's a disaster in the making. In any case, the IT department should know what to expect in terms of performance when the user load on the backup machine reaches its mean and mode.

Performance and Service-Level Agreement Information

Performance agreements are now widely used to set acceptable levels of service for internal and external system users. The service-level agreements (SLAs) that exist within these performance agreements provide a definition of acceptable service, outlining performance metrics like system response time and uptime, ways to manage problems, methods of handling disasters, and more. If the terms of an SLA are ignored, penalties can be imposed or the agreement can be terminated. SLA compliance falls squarely on members of the IT department. These quantifiable parameters should be clearly posted in your HA operations document.

Regulatory Requirements

The body of law that addresses sustained high levels of readiness and system resiliency is growing. And, because questionable business practices that cost investors and employees billions of dollars slipped under the radar for years, protecting data from "unintentional" destruction is now mandated for publicly held companies. Specific industries are subject to other forms of legislation. Stiff penalties can be imposed for failure to comply with these regulations.

Succinct documentation on these regulations and how they specifically apply to HA can offer important guidance to those who manage or modify the system. None of this information will really matter in the dark moments after a crash, but once access to the system has been restored, measures can be taken to bring the system back into spec.

Failover and Failback Strategies

What if? What will we do in specific situations? How much coverage will we have? Volumes have been written on failover and failback strategies. Ultimately, specific circumstances should trigger fixed responses. You can choose not to failover (cold failover) after being offline for two hours at midday, but your users and business will suffer. A hot failover strategy would have system users logged back on in minutes—but at a higher price. Any policies pertaining to how technicians are expected to react should be clearly stated, reviewed, and understood.

Changed System Values

There is a set of roughly 150 security-related system values in i5/OS. HA solutions reset some of these values upon installation, and some vendors offer a tool that documents these values both before and after they are changed. In the absence of such a tool, you can print a listing of these values. If you choose to print them, you should produce both before and after copies so that you can compare the values on these two lists if you find yourself in a situation in which you or other users cannot access the backup system.

Threats

Make a list of possible threats and estimate the likelihood of their occurrence and the impact they will have on business. By documenting the various types of threats that your production computing environment is most likely to face and an appropriate response for each one, you can limit vulnerability and speed up recovery.

Most threats can be placed in either of two categories: internal or external. Internal threats are posed by failing systems components and by employees. You can minimize your vulnerability to internal threats through sound system maintenance and tight security policies. External threats that involve loss of power or communications are relatively easy to circumvent with backup power supplies and contingency satellite uplinks. Other events like fires, floods, earthquakes, and tornadoes can be much more severe and represent a greater challenge for the IT department.

Sometimes, events occur that are so severe and widespread that recovery, while possible in some limited way, does not make sense. Reestablishing online order entry, SOA-based credit card transaction systems, and warehousing systems is futile if the warehouse and your entire inventory are in splinters. Understand and document what this limit is.

Network

Include documents on how different devices are connected to your network and how the network itself is configured. Do you have separate communications links for these backup systems at the site? Are both the primary and target systems on the same subnet, or are they on different subnets, each with its own IP addressing scheme? Know which IP addresses are hard-coded into programs. These cause a problem and are increasingly used in Web-based applications.

Be sure to also document how the individual user connections are established. Are they handled within a DNS server? Do they have hard-coded client access sessions with the IP addresses coded into the connection?

Hardware

Relative to hardware concerns, most companies use several different servers to satisfy their business computing requirements these days. In a failover situation, it's important to know which of these systems are duplicated at the backup site. Do you, for instance, have a backup fax server or Web server available to switch over to? Keep track of the systems that are available to you so you don't overlook something important and don't waste time trying to restore something you don't have backup assets to cover.

Software

The last thing you'll want to have to deal with in the midst of a disaster is a software license key issue. Make sure you have one license key for each copy of software that you have on the backup machine, and reference these keys in your operations document. Vendors are protective of their license keys and require help desk technicians to go through strict procedures to issue or reissue them. You don't want to have to deal with that when your production system is down and the only thing keeping users from signing onto the backup system is a software license key.

Also, know whom to contact, from your technical support representative to the salesperson who handles your account. The ideal situation is to have 24x7 software maintenance contracts with all of your vendors if you are a shop that works around the clock. Take a few notes on who will be available after normal working hours and who will not.

Database

A common practice used to simplify the rollover process in HA is to make the machine-specific local relational database names match on both the primary and backup machines. That way, it's transparent to the ODBC or JDBC drivers. If your primary box fails and you have to change the DNS entry on the network to redirect users to the backup machine, the common relational database name is automatically picked up. If different relational database names are used, you'll have to recompile or change every entry that points to the old relational database name. Make sure the common local database name is present in your HA operational documentation. If any confusion arises after the DNS entry is changed, you'll have the relational database names right in front of you.

Testing the HA Environment

You'll want to test your HA capabilities with a fair degree of regularity. You'll also want to test them after small problems occur and are resolved, because changes to the HA environment could adversely impact your ability to switch users to the backup system if a serious event occurs.

One quick way to test HA readiness is to initiate a role swap, cross your fingers, and see what happens. A test role swap verifies the viability of the communications link, the backup hardware, and the software applications. The test role swap only verifies basic functionality, though. You also need to evaluate or test the integrity of the data that resides on the backup machine. Is it up-to-date? Are all of the key relationships intact? During test role swaps, users typically stay on the backup machine for only a short time, not long enough to make sure that the data is 100% viable. For this, a small group of test users from different departments should be assigned the task of periodically conducting extended tests on the data.

Who Is Your HA Vendor?

If you're in a jam and can't seem to figure out what to do next, your HA vendor can help isolate problems. Chances are, the person you speak with will have worked through hundreds of failover conditions. Have contact names, emergency telephone numbers, and email contact information readily available.

Document Number Two: The HA System Log

In addition to the operations guide detailed above, another useful document that should be created, maintained, and frequently reviewed is a log used to record several types of important events that pertain to the HA environment.

Failover Events

Whenever the production system fails, it's important to log as much information as possible about why and when. If system variables change in the failover state and things don't match once recovery measures have been taken, you can go through your log, determine when the system failed, and restore these settings to the values that existed just before the failure.

Corrective Actions

Keeping track of who took corrective actions, what they did, and when these actions were taken is valuable for auditing purposes and might be required by law. Granted, when you are scrambling to restore sign-on screens, you're not taking thoughtful, introspective notes. You can't ask a firefighter to take incident notes when he has an axe in one hand and a hose in the other. When the smoke clears, though, it's important to make notes about every corrective action that was taken and what did and did not work.

Computing Environment Changes

It's good to have historical information on environmental variables handy. You want your backup environment to match your primary production environment. Most of this information is stored in proprietary IBM libraries and doesn't get replicated over to the backup system because it is release-dependent.

Exceptional Activity

Creations, deletions, and other security-related activity is logged in the QAUDIT journal. All you have to do is turn on auditing. If you have configured your system's security mechanisms appropriately, information on who changed what and when will be automatically logged.

Many HA solutions also run a scheduled audit and automatically log exceptional events that are specific to HA, like journals that are out of sync between the primary and backup machine, data that failed to transfer to the target machine, and intermittent communication failures.

The Devil Is in the Details

The more detailed your HA documentation is, the more helpful it will be. If the information is organized logically, anyone who is at the helm at the time of failure can use it.

Robert Gast writes for Vision Solutions, a leading developer and integrator of iSeries high availability and continuous availability solutions. Vision Solutions recently joined with iTera, Inc. to form the number one HA company in the world. Together, they specialize in the development of software solutions to increase and improve the availability of IBM System i data management systems and streamline data management processes.


BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: