In times past, the typical IT shop worked from 8:00 in the morning until 5:00 in the afternoon, at which point all the workers hit the local bar, drowned their data processing troubles in frothy beverages, and came back the next day to do it all over again. No one really gave much thought to the availability of his AS/400 after hours, except for the need to run nightly batch processes. But, as Bob Dylan said, “The times, they are a-changin’.”

Today’s IT shop must have its AS/400 available 24/7 to users around the world. As an IT manager, you have to ensure that your operations will continue, even during unplanned system downtime. One way to accomplish this is through data mirroring and data replication and by having an alternate system on hand to handle failover when the primary system goes down unexpectedly (unplanned outages) or when you need to take the primary system offline for scheduled maintenance (planned outages). There are many vendors out there who will sell you high-availability software and the expertise to implement it (see the Midrange Computing Yellow Pages at www.midrangecomputing.com/yellowpages), but perhaps you want to implement some of the more common techniques in your own shop, using your own software. If you fit into this latter category, this article is for you. It will get you thinking about what you and your staff need to do to implement high availability on your AS/400.

Defining High Availability

When I talk about high availability, I’m not only talking about the need to keep your AS/400 running. I’m also talking about the need for your applications, such as order-entry or Web-based item-master-inquiry programs, to continue to be available to your users, even during an unplanned downtime. This means that your critical data must be available as well. And when I talk about an unplanned downtime, I’m referring to any event, catastrophic or not, that physically prevents your users from accessing their data and applications on the primary AS/400. What is required then for high availability is another AS/400 that will step in and take over when the primary system goes down.

Although you could consider a single AS/400 running multiple instances of Logical Partitioning (LPAR) to be an alternate system, it’s likely that whatever was catastrophic

enough to knock out the primary LPAR is also likely to prevent access to all of the others. That’s why, for true high availability, you really need to have a second AS/400.

The Components

When it comes right down to it, there really aren’t many things required to implement high availability in your shop. In fact, the number of pieces you require is small, as you’ll see in this list of high-availability items and functions:

• A second AS/400

• Both AS/400s running OS/400 V4R4 (to take advantage of V4R4 clustering services)

• A connection (physical, such as one using OptiConnect and a system bus, or virtual, such as one connecting over a TCP/IP network or even the Internet) to the alternate AS/400

• Journaling of the data files you wish to replicate on the alternate system

• Commitment control in your applications to maintain data consistency across systems

• A few high-availability APIs to engineer application consistency during failover

That’s really all there is to it. And, truthfully, if you are not looking for the latest and greatest means of accomplishing high availability that take advantage of IBM’s clustering services, you can even skip the second and last bullet points on this list. Of course, implementing what is listed here may require some major reengineering of your connectivity and application architecture, which is why few shops go to the trouble of implementing it. However, it’s not too monumental a task, as you’ll see. So let’s take a closer look at the components I’ve listed and see what it takes to implement them. After that, you can decide whether or not you want to develop a solution on your own or purchase one “off the shelf” from one of the many high-availability vendors out there.

Making Sense of the Pieces

Obviously, the concept of a second AS/400 is self-explanatory. You need a second system, preferably somewhere off-site, that can be used in case something happens to the primary system or in case you want an alternate system to handle the workload of planned outages. If you want to take advantage of IBM’s clustering services, which allow you to create a “live” cluster of systems, you’ll need to ensure that both AS/400s are at V4R4. Clustering services consist of low-level operating-system code to provide the clustering and the APIs and exit points your applications can use to replicate data from one box to another and maintain application resiliency during failover. We’ll cover the concept of clustering and how to use it later in this article.

The next piece to consider is the connection between AS/400s. Depending on your current connectivity architecture, this is either a piece of cake or a really big deal. For example, if you are working in a shop that still uses twinax as the primary connection medium, you may want to consider upgrading to either Category-5 wire or fiber optics for increased reliability and speed. When you mirror data between systems, be certain that the data reaches the alternate system in a timely manner and that it is not corrupted during transmission. Implementing a high-availability solution is not a trivial matter, and the data conduit should not be considered trivial, either.

Your connectivity options for high availability can be almost any connectivity medium supported by OS/400:

• Twinax (not recommended)
• Ethernet
• Token-Ring
• TCP/IP
• Wireless
• OptiConnect/400

You’re probably familiar with all of these, except for maybe OptiConnect/400. OptiConnect/400 is an IBM product made up of hardware and software that allows multiple AS/400s to be connected by a high-speed fiber-optic bus. You must purchase a license from IBM to use this product. (For more information, see the OptiConnect/400 home page at www.as400.ibm.com/service/opticonnect.htm.) Basically, OptiConnect/400 provides you with the fastest, most consistent means of moving your data from one system to another. The downside is that both AS/400s must be in physical proximity to each other to use this service. If you want to ensure that an AS/400 system will be available to your users in case of a catastrophic failure to the primary box, you may want to connect your systems by using an Ethernet network or some other WAN topology. But if your main concern is having an alternate system available to your users during planned outages, you may not care whether or not your AS/400s are in different locations. Either way, your connectivity choices are fairly broad. Before making a final decision, you need to consider the amount of data you’ll be replicating between systems as well as the means of routing your users’ terminals/connection clients to the alternate system if the primary fails.

Remote Journal and Commitment Control

To maintain a consistent database—that is, one that contains the same data on each system—you must use the remote journal APIs to propagate database changes from the primary system to the target system. As additions or deletions are made to a database on the primary system, those actions are recorded in a journal receiver on a remote system. Most of the activity occurring via these APIs happens at the Licensed Internal Code (LIC) layer of OS/400. This provides better journal-entry-replication performance and allows the remote system to handle a larger part of the replication overhead as well as provide overall improvements in system performance.

To implement remote journaling using your own databases isn’t too complex, although it may involve a fair amount of work. All you really need to do is duplicate the databases you want to replicate on the alternate system and then modify the programs (RPG, CL, COBOL) that write to or delete records from these databases to also issue calls to the remote-journal APIs. Check out the IBM manuals OS/400 Backup and Recovery V4R4 and OS/400 Journal and Commit APIs V4R4 for more information on how to

implement this technique.

Commitment control, which is a concept all of your applications should be taking advantage of, can be considered the other half of remote journaling. Commitment control, in a nutshell, is the ability of a DB2/400 database to either “commit” or “roll back” database transactions. Its most useful function is the commit phase. By using this function, an application program lets a user make page after page (screen after screen) of changes to a database. These database changes are not physically made to a database until a commit operation is issued from the program. If the program terminates abnormally or if the logic of the program dictates that some grievous error has occurred, it can back out those database changes, which returns the database to the state it was in at the last commit operation. This is what’s known as a commit cycle rollback and ensures that database

changes are not made unless all the T’s are crossed and all the I’s are dotted, or, in IT terms, unless all database records have been synchronized.

In typical AS/400 shops, you almost never see applications written to use commitment control. I’m not sure I understand why this is, except that perhaps there is a perception that AS/400 databases are more stable, which is probably true, or that commitment control is an outmoded, unnecessary concept, which is not true. For whatever reason, commitment control is not being used as it should be. In your high-availability environment, you must ensure that your applications are using commitment control so changes made on one system are also made on the other. There’s a reason for this.

Imagine what would happen if, on your primary system, the application program managed to update only three out of five dependent databases before the primary system crashed. If those database changes were not executing under commitment control, the first three changes would be made on the secondary system just as they were on the primary. However, now the secondary system is left with two dependent databases that are not synchronized with the previous three. The next time an application needs to access the records from these unsynchronized databases, an error will result because information will be missing. Under commitment control, when the first system went down, none of the databases would have been updated. Instead, because it was a partial transaction that was never committed, all changes to the database would have been rolled back. On a single AS/400, commitment control is very important. In a high-availability, data-replication environment, it is imperative.

Clustering

For the best means of high availability, you should look into clustering. IBM’s clustering services, which are freely available in V4R4 of OS/400, allow you to integrate two or more AS/400s into a single logical environment. In the simplest cluster scenario (Figure 1), one AS/400 acts as the primary node, while the other is the idle standby node. The latter box basically just sits there, letting the first box replicate data and other resources to it. Every 3 seconds, the primary system sends a “heartbeat” to the standby system that is monitoring it. If the standby system fails to detect this heartbeat after four attempts, the standby assumes the primary system has failed (because of either a system crash or a communications failure or some other type of failure) and implements its failover procedures. It is at this point that the standby system comes alive. Interactive and batch jobs that were running on the primary system are restarted on the standby system, and other critical communications are routed to the standby box’s resources. When the primary system fails, the users’ screens go blank. When the standby system takes over, users are presented with an AS/400 logon screen. Now the users sign onto the standby system (which appears to be the primary system to the users), and they are immediately taken back into their application at the point where they left off.

Of course, to allow users to log right back into the same application at the point where the primary system failed, your applications have to be ClusterProven, a term coined by IBM to identify applications that provide this type of application and data resiliency. Although you don’t necessarily have to go through the entire IBM ClusterProven certification process for your in-house applications, you do need to use the cluster APIs at least to achieve that functionality. These APIs, along with exit programs, allow your applications to communicate with the cluster services code built into the LIC. (The list of these APIs and how to use them is rather long, so I won’t cover it here, but MC will detail this information in an upcoming article, so stay tuned. You can also read about these on your own in the IBM manual OS/400 Cluster APIs V4R4.)

Questions, Questions

Will your shop implement a high-availability environment? It really depends on the needs of your business as well as the resources (both financial and personnel) available to you.

Ask yourself these questions: Will your business be able to survive the downtime required for scheduled maintenance? How about the downtime associated with an unplanned outage, such as a fire? Do you have the programmers on staff or the funds to pay for consultants who can modify your critical applications to take advantage of commitment control and remote journaling? Is clustering the way for your business to implement the most secure and available systems? And perhaps most important of all, can your business afford to purchase and maintain the additional AS/400s that high availability requires? Implementing high availability in your shop is very “doable,” but not for the faint of heart. If, after reading this, you like the concept but hate the thought of in-house implementation, don’t despair. There are many quality vendors out there who can assist you in making high availability a reality in your shop.

References and Related Materials

• ”Clustering,” Mike Snyder, www.as400.ibm.com/beyondtech/clustering.htm
• IBM High Availability & Clusters home page: www.as400.ibm.com/ha/ index.htm
• IBM OptiConnect/400 home page: www.as400.ibm.com/service/ opticonnect.htm
• OS/400 Backup and Recovery V4R4 (SC41-5304-03, CD-ROM QB3ALE03)
• OS/400 Cluster APIs V4R4 (SC41-5886-00, CD-ROM QB3API00)
• OS/400 Journal and Commit APIs V4R4 (SC41-5882-03, CD-ROM QB3APE03)

Primary System

Idle Standby Cluster Node Cluster Node

Cluster

Data

Figure 1: Clustering allows multiple AS/400s to act as a single, logical system.