As a kid, I remember having Dominos around the house to play with. I don’t think I ever really “played” Dominos, but I would build things with them, and inevitably something would cause them to fall. As they came down, I knew before the last one fell exactly what I needed to do to rebuild them the next time.

Dominos isn’t just for kids anymore. In fact, Lotus Domino has become the buzz in the IBM AS/400 world, as it’s a great tool for business. To support Domino in the business world, administrators need a plan for rebuilding or recovering Domino’s components. In this article, I will discuss various mechanisms available to create an application backup strategy for Domino for AS/400.

Domino backup strategies should be defined to allow for recovering either parts of a Domino server, such as mail files or logs, or the entire server, in the event of a disaster. Domino is made up of programs and customization information that comprise the server, as well as databases that contain user data. On the AS/400, the programs and customization information reside either in libraries or in AS/400 Integrated File System (AS/400 IFS) directories, which contain more customization files, product information, and databases. Figure 1 contains a list of the libraries and AS/400 IFS directories that make up the Domino product.

When the server is initially created, it is important to create policies for safeguarding the location of files in the Domino environment. To have Domino files outside of the appropriate directories not only adds complexity to the backup strategy but also confuses the server. The Domino server understands from the set up what the name of the Domino data directory (where Domino user data files reside) will be. If, for example, /notes is the main directory for the server, this directory is available for the Domino server to work with. If a new file were created outside of this data directory, you could have security challenges and pointer problems within the server. Good policies for file placement are strongly recommended. If you are not sure about placement, Lotus reference materials, such as the one listed in the References and Related Materials section at the end of this article, or the Lotus Web site or support lines are good resources for answers.

Creating a Backup Strategy

Understanding what comprises Domino for AS/400 and where those library and AS/400 IFS directory components are located is the key to creating your backup strategy. When Domino is installed, you should immediately save a copy of the application (programs and customized files). Consider this your “golden” copy, at least until after the next release of Domino is loaded. The two main commands for saving Domino are the SAVLIB and SAV commands. Typically, Domino databases in the /notes/data directory change daily, and to capture those changes, the SAV command is used. Since Domino application library objects are not generally updated daily, the libraries can be saved less often using the SAVLIB command, with perhaps the only exception to this being the QUSRNOTES library. The Domino libraries and the AS/400 IFS *IBM directories will be captured during a full system save (GO SAVE Option 21), and should be saved individually prior to a Domino for AS/400 upgrade. Figure 2 (page 60) contains a list of the different methods for backing up Domino for AS/400.

Some AS/400 shops may have a very narrow window of opportunity for a daily save of AS/400 IFS directories, including Domino user directories. In this case, a partial or incremental backup should be considered. A partial backup may include saving only the mail and other critical databases on the server. Capturing only the mail files involves slightly modifying the backup string to include the mail directory and possibly the file extension:


SAV DEV('qsys.lib/tap03.devd') OBJ(('/notes/data/mail/*.nsf')

('/notes/data/mail.box')) SUBTREE(*ALL)


Saves all files in the /notes/data/mail directory which end with *.nsf


SAV DEV('/qsys.lib/tap03.devd') OBJ(('/notes/data))

SUBTREE(*ALL)


Saves all files in the /notes/data directory.

The MAIL.BOX file contains inbound mail, so if only saving mail files, it also may be included in the aforementioned backup string. Other examples of files to consider when taking this approach are shared mail files, discussion databases, or databases specific to certain applications. The obvious drawback to specifying individual files is that it introduces the possibility of missing or leaving out critical database files.

Another way to reduce backup time is by using incremental backups. The simpler of the two is the strategy of backing up any changes made since the last full backup. If, for example, you run a full backup on Sunday nights and all other nights you run an incremental backup, you need only restore the Sunday night backup and the latest incremental backups in the event of a crash. If you were to crash on a Thursday, you would restore Sunday night’s backup, then Wednesday night’s backup. With this strategy, backups grow larger throughout the week, but the recovery is fairly simple.

The following is an example of how to backup incremental changes since the last full backup and run each day between full saves:

SAV DEV('qsys.lib/tap03.devd') obj(('/notes/data'))

chgperiod(*LASTSAVE) subtree(*all)

If even this requires too much time, the incremental backup process can be taken a step further by only backing up changes since the last incremental backup, which requires fewer tapes and less time. The trade-off for saving time is recovery complexity: Now if you crash on a Thursday, you need to restore Sunday, then Monday, Tuesday, and Wednesday.

The following is an example of how to backup data from one incremental to the next. Notice that the start date on the change period parameter is the date after which objects that have been changed are to be saved.

SAV OBJ((‘/notes/data/*.*’)) CHGPERIOD(‘01/15/01’ 030000)

updhst(*yes)
SAV OBJ((‘/notes/data/*.*’)) CHGPERIOD(‘01/16/01’ 030000)

updhst(*yes)

This is run every night, updating the date and time each night until the next full Another consideration may be directory synchronization objects. Directory synchronization can be used to keep the Domino directory (also known as the name and address book) synchronized with the AS/400 System Distribution Directory (SDD). If using directory synchronization, consider backing up the AS/400 SDD files as well. These can be saved with the SAVOBJ command as follows:

SAVOBJ OBJ(QAOK*) LIB(QUSRSYS) OBJTYPE(*FILE) DEV(DEVNAME)

Remember that to prevent locks, directory synchronization should be stopped before backing up files. A call to the program supplied with Domino for AS/400 will take care of this for you. The call is run as follows: call QNOTESINT/QNNDIEND.

Timing Is Important

Now that you know what options are available, timing is of the essence. Most environments will require that the server be brought down to obtain clean Domino backups. There is an option for save-while-active in the SAV command; however, it is not bulletproof, and Domino generally doesn’t play nicely with it. The purpose of using the save-while-active function is to allow users to access Domino while it is being backed up; however, databases within Domino in use at the time of the save will be locked by the Domino application and therefore cannot be saved. And, no matter how many times you flush out the Domino server with the drop all and dbcache flush commands at the server console, the server is subject to users and tasks reinitiating their connections, creating more locks on files and objects.

Prior to ending the server before saves, there are maintenance jobs that run on the server that should be taken into consideration. Domino behavior is partially defined by a file in the user data directory called notes.ini. Some of the parameters in this file instruct the Domino server when to perform various types of maintenance. Once this maintenance is started, you should allow it to finish before bringing down the server for backups or other maintenance.

To look at this file, IBM initially provided a PTF to install the Edit File (EDTF) command. With later releases of Domino, this command was added as Option 13 on the Work with Domino Servers (WRKDOMSVR) command. In either case, you can look at

save.

the notes.ini file by issuing the following command from a command line: edtf ‘/notes/data/notes.ini’.

You are looking for statements similar to the following:

ServerTasksAt1=Catalog,Design,UpdAll
ServerTasksAt2=Object Collect mailobj.nsf
ServerTasksAt5=Statlog,Object Info -Full
ServerTasksAt6=Statlog

The lines above indicate that maintenance is being performed at 1:00 a.m., 2:00
a.m., 5:00 a.m., and 6:00 a.m. With this in mind, 3:00 a.m. might be a good time to schedule the server to come down for backups. In addition, you should time the backups to occur when the server has the least amount of user activity. You can manually check user activity by issuing the sh users command in the Domino console which will display users who are logged into the console and how many minutes since their last activity.

If agent managers are in use, you may want to ensure your backup doesn’t occur while a scheduled agent is running. To avoid such conflicts, you can use the Domino console command Tell amgr schedule to display all agents scheduled to run by the agent manager for the current day. Determine from the list of run times whether or not you will have any conflicts between the backups and the agent manager tasks.

If your environment dictates the highest availability possible, there are other alternatives. Clustering Domino servers lends itself nicely to a high-availability environment, allowing one server to be backed up while another services clients. Partitioning servers with replication also provides redundancy of data for backups. IBM’s Backup, Recovery and Media Services package is the first such application to support a true save-while-active backup, and Domino reportedly works well with it. If these options are out of your price range, backing up to save files on DASD is a last resort for saving time. Although the files must eventually be written to tape, write time to disk is generally faster than writing to tape; hence server downtime is lessened. Save files on disk can be written to tape at any time before the next backup. Before implementing this, just make sure the DASD for the save file is available.

Backup Tapes

Now that you have looked at backup types, schedules, and ways to limit the impact of backups to users, plan for how long the company needs to keep those backup tapes. Backup tape and file retention is something often planned for when running nightly, weekly, monthly, and yearly saves. You are most likely aware of the legal issues involving the retention of financial, medical, and other data that may be required to be produced or reproduced at a later date. The retention of email data is at the company’s discretion; however, you should take time to consider the importance of the data contained in each user’s mail file. Could there be a need to reproduce the content of an email message or one of the many database files sometime in the future? Consider the importance of any email that contains, for example, contract or medical information. Do you need to keep this information longer? Careful planning may prevent restore headaches in the future.

Executing the Backup

Once everything is in place, it is time to execute the backup. Figure 3 (page 61) shows a sample CL program for shutting down and backing up a typical Domino for AS/400 system. (Note, however, that you likely will need to modify this program to accommodate your own installation.)

The sample CL program outlines the basic steps for attempting a controlled shutdown, backup of user directory data, and restart in an unattended mode. There are lengthy delays built in to ensure that the server comes down clean; if, however, all else

fails, the server is brought down hard with an *IMMED. This forces an end to the Domino server job, tasks, and subsystem, but it should be used only as a last resort. With the server down, a backup of the /notes/data directory is executed, the subsystem description deallocated, and the Domino server restarted. Remember, to be successful, your program should be tested and retested to ensure it functions as you expect it to. Different systems may require longer or shorter delays, or additional shutdown steps. The goal is to bring the server down clean, get a good backup, and restart the server—with a minimal impact to users.

Although the information in this article may seem like a lot, with a little planning, backing up Domino for AS/400 is very straightforward. As you can see, Domino is not yet at a point where you can backup and restore individual documents within Domino databases from the AS/400 command prompts. Eventually, Domino may get to this level; until then, you can take satisfaction in knowing that you have taken all of the necessary steps to protect and recover all of the pieces in the event that Domino falls.

REFERENCES AND RELATED MATERIALS

• Lotus Domino for AS/400: Installation, Customization, Administration (SG24-5181-00)
• Lotus Domino on iSeries home page: www.as400.ibm.com/domino
• Lotus Support home page: www.support.lotus.com/lshome.nsf

Libraries and AS/400 IFS Directories 5769LNT Optional Part Backup Category Description

QNOTES *BASE *IBM Domino for AS/400 Product Library

QNOTESINT 1 *IBM Directory Synchronization Library

QNOTESAPI 3 *IBM C APIs Library

QNOTESCPP 4 *IBM C++ APIs Library

QNOTESLSKT 5 *IBM LotusScript Extensions Library

QNOESHTST 6 *IBM HiTest APIs Library

QUSRNOTES *ALLUSR Customization Information Library

/QIBM/ProdData/Lotus/Notes *IBM Product Information Directory

/QIBM/UserData/Lotus/Notes *ALLUSR Customization Information Directory

/notes/data *ALLUSR Directory for Databases on the Server

(created during server setup and named by the administrator)

Figure 1: Components of Domino for AS/400 include both libraries and AS/400 IFS directories and files and contain product information, as well as customization information.

Backup Method Description Recommended Use

GO SAVE, Option 21

GO SAVE, Option 22

SAVLIB *NONSYS

SAVLIB *IBM

SAV OBJ((‘*’) (qsys.lib *OMIT) (qdls *OMIT))

SAV OBJ((‘/notes’) (‘/QIBM/UserData/Lotus’))

SAVOBJ((‘/notes’) (‘/QIBM/ProdData/Lotus’) (‘/QIBM/UserData/Lotus’)

Restricted-state save. Saves the entire system. Save all QNOTESXXX libraries, QUSRNOTES library, and all AS/400 IFS directories (*IBM and *ALLUSR). This is a long- running save.

Restricted- state source saves *IBM libraries and directories on the system, no user data. This includes all QNOTESXXX libraries, as well as the /QIBM/ProdData/Lotus/Notes directory. No user data is captured with this save.

Restricted-state save. By itself, this save captures all libraries on the system (*IBM and *ALLUSR libraries). For full recovery of Domino, a SAV of IFS or specific Domino directories needs to be done as well.

Saves all *IBM libraries. A backup of QUSRNOTES, as well as SAV of AS/400 IFS directories, also needs to be done for Domino recovery strategy.

Captures all AS/400 IFS directories, except QDLS folders and QSYS.LIB folder. All Domino directories are saved; no Domino libraries are saved.

Weekly, monthly or quarterly, or before any major software application upgrades, hardware upgrades, hardware moves, or extended system outages.

Periodically as part of a full system save, or daily as stand-alone AS/400 IFS backup. This backup captures all AS/400 IFS directory and file data, except QDLS and QSYS.LIB information.

Captures Domino
user
AS/400 IFS directories. As part of a daily backup strategy for the Notes environment, this backup captures Domino user data.

Captures all Domino-related AS/400 IFS directories.

Before a Domino upgrade, in addition to a SAVLICPGM 5769LNT and a SAVLIB QUSRNOTES, this captures the IBM Domino directories, as well as user data.

Figure 2: Depending on the type of information to be saved, there are a number of backup options available.

Either before applying system PTFs or as part of a regular backup strategy, in addition to an *ALLUSR or a SAVLIB strategy.

Weekly, monthly, or quarterly as part of a full system save strategy. Use in conjunction with SAVSYS, SAV, and SAVDLO. Before any major application upgrades, hardware upgrades, hardware moves, or extended system outages.

Either before applying system PTFs or as part of a regular backup strategy, in addition to an *ALLUSR or a SAVLIB strategy.

5769PW1 V4R4M0 990521 SEU SOURCE LISTING 01/22/01 10:39:45
PAGE 1

SOURCE FILE . . . . . . . JULIE/QCLSRC

MEMBER . . . . . . . . . ENDDOM

SEQNBR*...+... 1 ...+... 2 ...+... 3 ...+... 4 ...+... 5 ...+... 6 ...+... 7 ...+... 8 ...+... 9 ...+...
0

100 PGM

200 CALL PGM(QNOTESINT/QNNDIEND) /* END Directory Synchronization */

300 MONMSG MSGID(CPF0000)

400

500 DOMINOCMD: SBMDOMCMD CMD('drop all') SERVER(DOMSVR1)

600 SBMDOMCMD CMD('dbcache flush') SERVER(DOMSVR1)

700

800 ENDDOMSVR: ENDDOMSVR SERVER(DOMSVR1) OPTION(*CNTRLD)

900 MONMSG MSGID(CPF0000)

1000 DLYJOB DLY(300)

1100 ENDSBS SBS(DOMINO01) OPTION(*CNTRLD)

1200 ALCSBS: ALCOBJ OBJ((QUSRNOTES/DOMINO01 *SBSD EXCL)) WAIT(300) /Check to see if the
Domino Subsystem is down */ 1300 MONMSG MSGID(CPF1002) EXEC(DO)

1400 ENDDOMSVR SERVER(HO3DM3) OPTION(*IMMED) /*END DOMSVR AND SBS */

1500 DLYJOB DLY(120) /*WAIT FOR FINISH */

1600 ENDDO

1700 DOMSAV: SAV DEV('qsys.lib/tap.devd') OBJ(('/notes/data')) +

1800 SUBTREE(*ALL) SEQNBR(1) ENDOPT(*LEAVE) INFTYPE(*ALL) UPDHST(*YES) +

1900 CLEAR(*ALL) /*SAVE NOTES DATA DIRECTORY AND ALL SUBDIRECTORIES */

2000 DLCOBJ OBJ((QUSRNOTES/DOMINO01 *SBSD *EXCL))

2100

2200

2300 STRDOM: STRDOMSVR SERVER(DOMSVR1)

2400 ENDPGM

* * * * E N D O F S O U R C E * * * *

Figure 3: Automate the nightly task of shutting down and backing up Domino for AS/400 with a simple CL program.