The AS/400’s reputation for high availability in the midrange marketplace has earned great respect in the industry. IBM estimates that the AS/400 supports around 99.9 percent availability—about 2 to 5 hours of unplanned downtime per year.

In any organization, the requirements for high system availability differ. The IT director needs business applications up and running 24/365, and all of the director’s customers need access to their data, regardless of the time zone they operate in or the hours they work.

To achieve the level of high availability required by IT shops and customers, the impact of a planned or an unplanned outage should be minimized. This can be accomplished by careful planning. High-availability software needs to be easy to administer so systems remain available and exposure is minimized. An enterprise also needs a carefully crafted system availability plan. The plan needs to address such things as backup power supplies and off-site tape recovery plans. Key elements of the system availability plan are the off-site tape location and the time required to return to a ready state after an unplanned outage. Test restores of data from the tape archive should be performed to test these times as many factors such as tape speed, database organization, and the number of logical views present can drastically affect the time required to restore data. Although most people think of unplanned outages as earthquakes, fires, and hardware failures, a large percentage of unplanned outages are due to operator and/or application errors. Regardless of the cause, when you have an unplanned outage, the ability to roll back to a particular save or checkpoint before the error occurred is critical. Proper planning can help you with all of these issues. Depending on your objectives, you may not need multiple AS/400s to accomplish your goals.

The Costs of Being Prepared for Outages

A solution to provide 24/7 access to all users across the enterprise need not require multiple systems. The concept of single-system replication (SSR) can accomplish many of the goals and availability requirements as other solutions that need multiple footprints. SSR allows access to query databases during backup windows that traditionally required the system to be in a restricted state. Also, batch runs can be configured to minimize the impact on the

users. With that in mind, here are some of the impacts that preventing outages may have on the users.

The “insurance” costs of preventing single-system outages and data loss are as follows:
• Loss of access to applications and data while backup operations are carried out
• Impact on performance and response time because of high-availability activity such as journaling and save operations
• Increased single-system capacity requirements

The costs of the actual incident can include the following:

• Loss of data and information
• Loss of access to the system during the time required for recovery time

Alternatives

Figure 1 reviews the available alternatives for single-system availability. These figures proceed along a continuum from relying on the native high availability of the AS/400 to the use of additional tools and utilities to reduce the backup window and to ensure less data loss. As you can see, there are many alternatives.

The traditional backup method is to save and restore data and objects to tape. The problem with this method is that it requires a dedicated backup window where files are not allocated to users because the backup commands require an exclusive lock on the objects being saved or restored.

An alternative to the save and restore method is to back your data and objects up to an AS/400 save file. Backing up to save files shortens the backup window. The downside to this plan is that your AS/400 will require additional DASD to hold all of the saved data. However, once the objects have been backed up to a save file, they can then be saved to tape in a separate step. Keep in mind that this method also requires an exclusive lock on the objects to be saved and restored.

A method that does not require an exclusive lock be placed on the objects to be saved and restored is the Save While Active technique. With Save While Active, you can save files and objects even if users are currently using them. It does, however, have a big limitation: If an open commit cycle exists, Save While Active cannot get to a synch point and will not allow the save. Commit cycles are sets of transactions, such as the results for an entire race in the Olympics. If only half the race results are in, the cycle is still open. This concept is important in data integrity as a failure without commitment control could result in half a race being available in the database and the wrong winner could be indicated. Save While Active needs to establish a synch point to which it can return to for each object. Open commit cycles prohibit Save While Active from getting a lock on the file and rejecting the save. As more AS/400 applications begin to use commitment control, because of the benefits it provides to journaling performance and recovery integrity, this will become even more of an issue.

Logging transactions to tape is another option that will improve single-system high availability. The inherent problem with backups is that you are only as current as your most recent backup. Tape logging software continuously writes journal entries to tape as they appear. The tape can be taken off site for disaster recovery purposes and reloaded at any time to data changed since the last backup. This does not improve recovery time but does eliminate lost data between backup windows.

Single-system Replication

Another single-system solution is SSR. SSR works by reading a source journal and applying it to target physical files located in other libraries or possibly in other Auxiliary Storage Pools (ASP) on a single AS/400. SSR software, regardless of whether you write it

yourself or purchase it from a vendor, should provide, at a minimum, the following features:

• Replicates data to another ASP or to other libraries.

• Automatically backs up data from either the replicated data or production data. This allows users and applications to access data while the backup runs on the other copy. The apply process of the journaled entries should stop while the backup is running.

• Provides a frozen version of the data.

This solution allows you to freeze the data so even though a nightly batch process may be changing the data, the users see a frozen version. This is important for applications that create data that might not make sense to users if viewed in the middle of the process.

• Provides the ability to redirect users and applications to a backup copy of your data if the primary ASP where the data resides fails.

Sample SSR Scenario

Whether you write your own SSR software or purchase it from a vendor, using it will generally follow a common process. The following is a sample scenario for using SSR. Many variations are possible.

1. Replicate data to another library or ASP on the same system.

2. At the beginning of nightly batch run, stop the apply of the journal entries to the replicated copy while journaling continues.

3. While the batch run is in process, make a backup from the replicated data.

4. After a batch run, the software should activate the replication process and then synchronize the replicated database.

Journaling

While journalizing on the AS/400 does not usually have a severe impact on performance, when an application uses long-running, single-threaded batch jobs, the impact can be significant. Because journaling plays an important role in high availability, it is important to optimize its use. To do that, you’ll first need to understand some of the more important aspects and uses of journaling, such as commitment control.

Normally journaling writes to the database’s journal are performed synchronously. This means that when a database record is changed, the entry is written to the journal before the application can continue. When an application uses commitment control, the writes to the journal are buffered and, basically, are written as a block when a commit or rollback is issued, resulting in more efficient performance.

Because IBM recognizes the need for better performance in a journaling environment, it released a new journal caching PRPQ. This PRPQ is available in beta to improve journaling performance. Basically, the PRPQ buffers write to disk for the journal without requiring commitment control. An API is available along with a call to the IBM program QJOSPEED. One benefit of the PRPQ is that it does not require any changes to applications. Contrast this to commitment control, which requires programmers to determine when to open commit cycles and when to commit them or roll them back.

One downside to the PRPQ is that it does not provide an easy capability to recover at a commit boundary. Commitment control is still a recommended technique for data

integrity in both a single-system and a dual-system high-availability environment. Nevertheless, the PRPQ is a convenient way to minimize the impact of journaling on your system.

Using Logical Partitioning

Logical partitioning (LPAR) has been available since V4R4 and allows a single system to be divided into multiple logical partitions. These partitions can be reassigned if requirements change but currently only by performing an IPL. High-availability software should define each partition as if it were a separate system and then mirror data between the partitions. In the case of a failover, users are connected to the backup partition. Of course, each LPAR system has a controlling partition and, if that fails, the whole system is down! Nevertheless LPAR provides a degree of fault tolerance for AS/400s. Maintenance and upgrades can be performed on partitions separately with the other partions active because partitions don’t even have to be at the same level of the operating system, although LPAR requires a minimum of V4R4. Backups can be done from copies maintained by High Availability Business Partner (HABP) software on secondary partitions in much the same way as described for SSR.

Figure 2 shows a comparative index of the benefits of utilizing various single-system high- availability techniques, using a relative impact index scale from 0 to 5. Within this figure
I’ve highlighted the following backup and availability techniques: tape backup and SSR.

The Five Impact Factors

Take a look at the chart shown in Figure 2. I’ve plotted the five impact factors for each high-availability solution. These are some common needs that most shops would experience.

Recovery Time

This is the time it takes to recover a system to an operational state after a failure for which data cannot be recovered. Tape backup has a higher recovery time than single-system replication. Single-system replication has a shorter recovery time because the databases are already built in an alternate library on the same system, and the libraries can simply be renamed or users can be pointed to those libraries.

Performance Impact

The performance impact of backing up to tape is minimal because no high-availability activities occur except during the backup window. Single-system replication reflects the need to activate journaling and then read and apply the journal entries.

Data Loss on Recovery

Backup systems have a high data loss on recovery because backups cannot be done continuously.

Site Disaster Risk

If only the SSR is used and the entire site is lost, then the data is lost as well. Of course, most single-system replication applications are combined with a backup strategy, and since single-system replication permits more frequent backups, no dedicated window is needed.

Benefits

System Use Impact

If tape backup is used, a suitable backup window must be provided during which users must be off the system. Single-system mirroring options allow backups from a copy of the data.

Options Abound!

Even if your business cannot afford an AS/400 cluster or live backup system, you can take steps to protect your valuable data in case of a failure. The concept of a single-system replication system can eliminate the time the backups keep users from accessing your system. For failures that do not involve the entire system, you can even provide immediate failover. Beyond that, you can also greatly improve the availability of your single system through the use of journaling.

REFERENCES AND RELATED MATERIALS

• AS/400 Performance, IBM Benchmark Center, Partners in Development, May 1999: www.as400. ibm.com/developer/cbc/as400perf/index.htm

• Beta Version, PRPQ PID 5799-BJC: http://as400service.ibm.com/supporthome.nsf/Document/10000035 (This is valid until December 31, 2000 and is available from IBM, Rochester. This is a recently released PRPQ that will become a PTF in the future.)

Solution Configuration Backup Impact Recovery Steps

Device mirroring, RAID Not discussed in this article

Standard tape backup Standard save (SAVLIB Requires backup Restore files or SAVOBJ) window

Use Save While Active Users can be on if Restore files no commit cycles are open; impacts
performance

Backup to save files Backup save files to tape Slightly shorter backup Restore files window; more disk
space

Plan use of Auxiliary Separate ASP for journal Journaling runs faster Can recover journal Storage Pools (ASPs) if journal ASP does not fail

Continuous logging Read journal and log Can run backups Recover last to tape to tape or another device less frequently backup and replay transactions since backup to database

SSR Backup from production Combination of backup Restore files version and no impact on users Site failure exposure

Backup from replicated Combination of backup Restore files or use version and no impact on users replicated copy

Site failure exposure

LPAR Replication between Similar to dual Switch to alternate partitions systems products partition, if available

Tools to reduce Commitment control Requires application Allows recovery at journaling impact changes commit boundaries

New journaling PRPQ No application No commit changes boundaries for recovery

Figure 1: Single-system availability solutions protect your data.

5

4

3

2

1

0

Tape Backup SSR SSR with Log Recovery Time Pefromance Impact Data loss on recovery Site disaster risk System use impact

Figure 2: This comparative index illustrates the benefits of utilizing various single-system high-availability techniques.