Part 2 of a 2-part series

A review of several recovery processes and the QUSRTOOL journaling commands.

In last month's article, I explained the concepts of journaling to you, and also discussed journals, journal receivers and journal entries. I also reviewed commands that you can use to retrieve journal entries from a journal receiver. But journaling is much more comprehensive than simply creating journals and receivers, and occasionally retrieving journal entries. In fact, one of the major design goals of journaling is to facilitate database recovery. This entails a number of complicated processes that you will have to become familiar with if you intend to use journaling for recovery purposes.

This month, I will review several recovery processes, with the idea that you will have to do quite a bit of additional research and planning in order to use those processes. I will also review the journaling commands that are included in IBM's free QUSRTOOL library.

Starting Journaling

Before you can use any of the journaling recovery functions, you must start journaling, using the following steps:

1) Create journal receivers with the Create Journal Receiver (CRTJRNRCV) command.

2) Create journals with the Create Journal (CRTJRN) command.

3) Start journaling physical files with the Start Journal Physical File (STRJRNPF) command.

4) Optionally start journaling physical and logical file access paths with the Start Journal Access Path (STRJRNAP) command.

5) Save all files that are being journaled.

You will also have to verify that you are performing each of these steps correctly. For example, you will have to assign correct names to the journal receivers and locate the journals in the same ASP as the libraries where the journaled files are located. Refer to the CL Reference and Backup and Recovery Guide for additional information about all of the factors involved in each step.

Although there aren't a lot of things that need to be done to start journaling, you will have to invest some time in research and planning before you start. As I mentioned last month, it might not be the best idea to simply journal everything. You need to be clear about why you are journaling: is it for database recovery? Security and auditing? Testing? Or, a combination of those and other factors. Why you are journaling needs to be kept in mind so that you will be able to manage the journals on your system.

Things You Can Do

Once you have journaling active, there are several things that you can do. In fact, if you will be using journaling as one of your primary database recovery techniques, you should create and run through several test scenarios, since the material that I can cover in this article is only an outline. Using journaling for recovery must be rehearsed as much as any other part of your disaster recovery drill; when the moment comes, it may be far too complicated to try to sort out journal recovery along with everything else.

The activities that I will discuss are described in more detail in the Backup and Recovery Guide. I mention these activities here so that you will have some idea about what is possible. In addition to these standard activities for which there is some degree of system support, you can always create your own programs to extract and use journal entries.

Forward Recovery

Forward recovery is used when you have a backup of a file, current journal receivers, and some degree of damage to the file. The file could be damaged for any number of reasons: power failure, disk failure, etc.

The simplest forward recovery technique is to use the system-provided processing. If you execute the Work with Journals (WRKJRN) command, the list display of journals includes Option 2, "Forward recovery." If you are near a terminal, run the WRKJRN command now, and select Option 2 for a journal. When you select that option, the next display is the Work with Forward Recovery display. This display shows a list of journaled files and members associated with the selected journal. Also shown is the status of the file, as far as the journal is concerned. If the status field is blank, then the file member and the journal are synchronized and usable. If the status field is damaged or not synchronized, then you can select Option 2 to apply journaled changes; the system will prompt you to restore either the file member or the journal receiver.

To recover a damaged member, the system will guide you through the following steps:

1) Dependent logical files are identified and deleted.

2) The file itself is deleted.

3) The file is restored from the most recent backup.

4) Dependent logical files are restored.

5) The Apply Journaled Changes (APYJRNCHG) command is prompted and executed.

6) The system prompts for the restore of the receiver if a journal receiver is required as a result of the APYJRNCHG command, and the receiver is not on the system.

You may be forming the opinion that this is quite a lot of work, as opposed to simply restoring the file. It is in fact a lot of work, and there is a great deal of accuracy required on your part to locate the proper objects on the save media. Only you can decide if this amount of effort is of value to your installation. If your files are easily recreated and made current, then perhaps the daily backup is sufficient. However, if you are in a highly volatile environment where a great number of transactions occur between backups, then journaling may be the only method you have to approach currency (short of mirroring, dual systems, etc.). You can also begin to understand why I suggest that you practice recovering with journals, if for no other reason than to expose the critical parts of this scheme where planning and resolve are required.

Backout Recovery

The opposite of forward recovery, backout recovery is useful when the file is not damaged (at least the system doesn't consider it to be), but there are transactions that must be removed. For example, a batch of transactions may have been posted to the wrong period or year, or transactions may have been written to the incorrect member. The system provides support for backout recovery on the Work with Journals display, Option 3.

You can prompt for that option and look at it now if you are at a terminal. On the Work with Backout Recovery display, you can select Option 2 to remove journaled changes for any of the files listed. When you use Option 2, the system prompts you for the Remove Journaled Changes (RMVJRNCHG) command. You can specify which journaled changes are to be removed; for example, you might be able to specify the exact job that created the changes which you want to remove. As with the forward recovery process, the system prompts you to restore any journal receivers that are required to complete the backout recovery, if those receivers are no longer on the system.

The most important point to consider about backout recovery is that it can only be used if you have both *BEFORE and *AFTER images in the journal for the selected file. You have to specify IMAGES(*BOTH) on the STRJRNPF command to get before and after journaling for a file. Obviously, if you keep before and after images, your journal receivers will be larger than if you only keep after images. So nothing is easy! Another tradeoff decision that you have to make. You might not need to plan for backout recovery for all of the files that you journal, or you might decide to keep both images for all of your journaled files, just to simplify things. Be aware of this requirement, and be aware of the increased storage for the journal receiver.

Recovering Damaged Journals and Receivers

Journaling is so intimately associated with recovery that the Work with Journals display even includes options to recover damaged journals and damaged journal receivers.

On the Work with Journals display, Option 6 is used to recover a damaged journal. If you select this option for a journal that is not damaged, the system sends you a message that the journal is not damaged. However, if the journal is damaged, the system starts a rather involved process to try to recover from the damage. The steps taken include:

1) Attempts to determine files that are being journaled to the damaged journal.

2) Ends access path journaling for all access paths journaled to the damaged journal.

3) Ends journaling for all files journaled to the damaged journal.

4) Deletes the damaged journal.

5) Shows the recover damaged journal display which allows you to restore or create the journal. If you create the journal, specify the receiver names.

6) Shows the list of files to be journaled. At this point, journaling is started for those files.

7) Shows the list of files whose access paths are being journaled. At this point, access path journaling is started.

8) Associates the receivers with the journal.

I don't know how to damage a journal, so I can't tell you how to practice recovering a damaged journal. Most of these steps are meant to be handled by the system, but it looks like you will want some additional assurances before relying on this. Some of the QUSRTOOL commands that I review in this article provide additional information about journals and journal receivers that may increase your comfort level when confronted with a damaged journal.

The system also provides for the recovery of damaged journal receivers, Option 7 on the Work with Journals display. Again, the system guides you through the recovery process, which seems to be simpler than recovering from a damaged journal. The steps that are taken include:

1) Runs the Change Journal (CHGJRN) command to detach the current damaged receivers. You indicate if you are changing to existing empty receivers, or if you want new receivers to be created. If you want new receivers, the CRTJRNRCV command is prompted.

2) Deletes the damaged journal receiver.

3) Prompts for the restore of the damaged receiver.

Once again, this needs to be taken on faith, since I don't know how to willfully damage a journal receiver and afford myself the opportunity to practice. Pulling the plug doesn't seem to cause the necessary damage--it merely increases the IPL time.

All of the preceding recovery activities should convince you that journaling is not a substitute for a conscientious back-up program. In fact, the back-up routine may be more complicated, since you will want to ensure that the journals and journal receivers are being saved and are easily identifiable. If you are not journaling, but it looks like you may start, you will have to allow time to plan for, implement and test these techniques. You could be very disappointed if you start journaling in a haphazard manner and invest your hopes in it as a recovery mechanism. You don't want to depend upon it, then find on the day of reckoning that you innocently forgot some detail which will prevent you from using it for recovery.

QUSRTOOL Journaling Commands

Every AS/400 is shipped with the IBM-supplied QUSRTOOL library. If you haven't installed the library and created the tools, but you are using journaling, you owe it to yourself to at least examine and create the tools that are used with journaling. These tools (commands) supplement the system-supplied commands, and will probably make your journal maintenance activities easier. There are nine commands in four broad classifications: journal and receiver analysis, journal maintenance, journal entry application and journal summarization.

Journal and Receiver Analysis

Three commands are included: Print Journal Summary (PRTJRNSUM), Convert Journal Attributes (CVTJRNA) and Convert Journal Receiver Attributes (CVTJRNRCVA).

Immediately after you start to journal, run the PRTJRNSUM command. This prints a summary of journals in one or more libraries, including information about each journal receiver and a count of the files and access paths being journaled. The information printed is based on the Work with Journal Attributes (WRKJRNA) command, the difference being that all of the information is in one place on a report, rather than on various displays.

The CVTJRNA and CVTJRNRCVA commands are used to create database file records of journal and journal receiver information. You can develop your own programs to query and print those records, and another of the QUSRTOOL commands, Resume Journaling (RSMJRN), depends on the output of the CVTJRNA command.

The CVTJRNA is based on the WRKJRNA command. Records are written to three database files:

JRNFILP - one record for each file being journaled

JRNACPP - one record for each access path being journaled

JRNRCVP - one record for each receiver known to the journal

The CVTJRNRCVA is based on the output of the Display Journal Receive Attribution (DSPJRNRCVA) command. This creates a record for a specific journal receiver that includes information about the receiver, including the current status, the attach date, number of entries and starting sequence number.

If you have been journaling, you may want to execute those commands, then run a Query report against the files. You may be able to use the information in the files for journal management programs that you create.

Journal Maintenance

Maintaining journals is a fact of life; you don't just start journaling and then walk away. Journal receivers are like newborn babies that grow and demand to be fed, and...well, there's one thing they don't do like babies. However, if you are journaling volatile files, and particularly if you are journaling access paths, the journal receivers will grow, and at some point you have to change to new receivers.

Another important journal maintenance consideration involves your schedule for saving the receivers. One of the interesting features of journaling is that you can change receivers while the journaled files are in use. This is a little like a relay race where a baton is passed--although there is some inefficiency in the passing of the baton (changing the receiver), the first runner can relax (the receiver can be saved) after the baton is passed. When receivers are changed in this manner there is no overlap in the journal entries; there is simply an ending entry in the first receiver that points to the next receiver, and the first entry in the next receiver points back to the first receiver.

The Maintain Journal (MTNJRN) command can be used to generate new journal receivers and delete old journal receivers. This command lets you specify the IBM-supplied journals individually or as a group, or you can specify your own journal. This command would typically be used as part of your back-up process: first run the MTNJRN command (which detaches the current receivers) to generate new receivers, then save the old receivers and run the command again to delete the old receivers.

You may never need to use the RSMJRN command, but if you need it, it can save you quite a bit of work. This command can be used if you decide to move a journal to another library (which cannot be done with the MOVOBJ command), or if you want to create a journaling environment on another machine that is a duplicate of the current environment. To understand why this is a useful command, follow these steps to compare what is involved in manually moving a journal:

1) End journaling: End Journal Access Path (ENDJRNAP) and End Journaling PF Changes (ENDJRNPF) commands.

2) Create the journal in the new library.

3) Restart journaling for each physical file and access path.

Although it doesn't look like much, you have to specify every file and access path to journal in the STRJRNPF and STRJRNAP commands (you can specify *ALL in the ENDJRNAP and ENDJRNPF commands). This means you need a listing, then you have to sit and type it in, etc.

Although the RSMJRN command looks like it's more involved, it automates the restart part:

1) Use the CVTJRNA command to capture the journal environment.

2) End journaling (ENDJRNAP and ENDJRNPF commands).

3) Create the journal in the new library.

4) Use the RSMJRN command (this uses the files created by the CVTJRNA command).

You can also use this command to resume journaling on another system by sending copies of the files used in the CVTJRNA command to the other system.

Journal Entry Application

There are three QUSRTOOL commands used to apply journal entries to database files: Summarize Journal Entries (SUMJRNENT) and Apply Summarized Journal Entries (APYSUMJRN) which are used together, and Apply User Change (APYUSRCHG).

SUMJRNENT and APYSUMJRN commands are used when you are performing a forward recovery to a file. With summarized journal entries for a given record in a database file, the only journal entry that needs to be applied is the last one associated with that record. This contrasts with the system-provided APYJRNCHG command, which starts with the specified journal entry and applies all journal entries through the specified ending journal entry. That command has no discrimination; if a record had several dozen journal entries in the journal receiver, all of those entries are applied in succession, even though the end result is the same as the last journal entry for that record.

The SUMJRNENT command is used to create a summary file that is used by the APYSUMJRN command. The SUMJRNENT command brings up journal entries from a journal receiver with the Display Journal (DSPJRN) command; those entries are placed into a database file. You must ensure that the file can hold the largest record that was journaled; the command as provided can accommodate a record up to 2000 bytes long (you can modify the record length if you need a longer record).

The APYSUMJRN command is used with the output file created by the SUMJRNENT command. When you use this command, specify the database file to which you want to apply summarized journal entries; apply summarized entries to one file at a time. If you use put direct operations to the database file, there are special considerations that apply with this command. (A put direct operation occurs when you write to a specific relative record number in the file, or for example, when you keep track of the relative record numbers of deleted records in a file and write new records to that location.)

APYUSRCHG is used as an alternative to the system-supplied APYJRNCHG command. APYUSRCHG can be used to circumvent some of the restrictions of the APYJRNCHG command. APYJRNCHG can only be used to apply changes to the original file; the file cannot be a copy in a different library. Also, the file must have been being journaled when it was saved in order for APYJRNCHG to work with a restored version of the file.

APYUSRCHG works with converted journal entries; that is, entries that have been brought up from the journal receiver with the DSPJRN command. You can use APYUSRCHG to apply journaled changes when APYJRNCHG won't work. For example, in the following cases:

Journaling was started, but the file was not saved. If the file is restored, journal entries cannot be applied with the APYJRNCHG command. The entries can be restored with the APYUSRCHG command.

Recovery of high usage files. For example, a file is open and in use 24 hours a day. You can take an occasional copy of the file and keep it current with the actual file, applying journal entries with the APYUSRCHG command (remember that APYJRNCHG does not let you apply journal entries to other than the original file). To recover the high usage file, you can rename or copy the duplicate file, and apply the last set of journal changes to it.

You need to maintain a duplicate copy of the file on another system. Rather than copy the entire file every time, you can simply send a file of the journal changes and apply them with this command.

Journal Summarization

Finally, there is a command in QUSRTOOL that you can use to summarize journal entries. This command is Print Journal Analysis (PRTJRNANL). It works with the header portion of DSPJRN records, and summarizes open, close, add, update and delete activities for journaled files. The summary can be prepared by member, program or user. This command can be used to help you understand and track usage of your database files.

Quite Involved, But Worth It

As I mentioned in part one, journaling is quite intricate since it involves so many aspects of the AS/400. Some of these usages include security, auditing, backup, recovery and programming. If you were to pick just one usage, you would probably find that journaling by itself does not completely address all aspects of that area of interest. For example, it is not enough to simply use journaling for auditing or back-up purposes; you must also use other system functions to implement and maintain those functions.

The advantage of using journaling--rather than creating your own programming-- is that journaling is simple to understand and implement, and all of the supported functions are well documented. As you may have realized from these articles though, journaling is not just a matter of starting it and letting it run. Once you start journaling, you must devote some time to it on a regular basis, especially if it is an integral part of your recovery plans. If you do plan to use journaling for recovery purposes, you should create and practice several recovery situations, documenting each step and noting any problems that you had. Recovery using journals can be far more complex than simple restores of saved objects; if you are to succeed in a recovery situation using journals, you must be able to access all of the required files, journals and receivers.

If you will be using journaling, examine the commands provided in the QUSRTOOL product, even if you don't plan to use those commands. The source for all of the objects used in the commands is provided, so you can study the techniques used and possibly modify the commands for your situation.

Next month, I will examine an additional use of journaling, which is very closely tied to application design and programming: the use of commitment control.

Also Read

Practical Middleware: Architecting for the Future

Practical Middleware: Starting the Journey

IBM i 7.2: The Carrot Versus the Stick