Learn about the V6R1 watch enhancement and understand a file's DLTPCT attribute.

This is the fourth in a series of articles on detecting that certain messages have been sent on your system and then making processing decisions based on those messages. The underlying technology, known as "watch support," became available with V5R4.

The first article, "One Approach to System Automation," introduced the Start Watch (STRWCH) command and provided the source for a user exit program to run when the local system time changed due to Daylight Saving Time transitions. The second article, "Handling System Changes Automatically," discussed the internals of that user exit program. The third article, "Re-enable Disabled User Profiles," provided further examples of the capabilities available with watches. If you have not previously read these articles, you should do so before reading the current article. In this article, we will look at how to detect when a file has exceeded the desired percentage of deleted records within the file and how to automate the reorganization of that file. Using watches, we can reorganize only those files needing to be reorganized and not have to interrogate, or poll, the other files on the system. We'll simply let the system tell us when a file is ready.

First some background. When you create a file using the Create Physical File (CRTPF) command, you can specify, with the DLTPCT keyword, the maximum percentage of deleted records you want in each member of the file. When the percentage of deleted records in a member exceeds this threshold, a message is sent to the QSYSOPR message queue and the QHST history log, informing you of the situation. As the default for DLTPCT is *NONE, and very few users have the time or desire to examine all of the parameters that are available on system commands, many users never see this message or even know that it exists. But by setting an appropriate value on each file--perhaps 10 percent for some files, 25 percent for others, etc.--the system can inform you of the need to reorganize the file in order to reclaim the storage associated with deleted records. The actual message sent is CPF4653 (DLTPCT parameter value for member &1 exceeded) with message replacement variables providing you with the file, library, and member name. And in case you're concerned that you may need to re-create your files in order to use this support, the DLTPCT keyword is also available with the Change Physical File (CHGPF) command (though CHGPF does require an exclusive lock on the file, so you may need to schedule a time for the CHGPF).

Before we look at the watch exit program for the CPF4653 message, we should discuss a few considerations related to this message. First, this message is sent whenever the file is closed and the current number of deleted records in the file causes the percentage threshold to be exceeded. For files that are used heavily and closed frequently, this can lead to a significant increase in the number of messages being logged to the QSYSOPR message queue and the QHST history log. For some users, there are already too many messages in QSYSOPR, so this approach--enabling CPF4653 messages--may not be for everyone. One alternative approach to using CPF4653 and still detecting files in need of reorganization can be found in the book IBM System i APIs at Work, Second Edition.

A second consideration is related to the processing of the CPF4653 message. Let's assume we have an initially empty file A with a DLTPCT value of 20. Further, we will schedule any file reorganizations to start at 2:00 a.m. each morning as we certainly don't want to do reorganizations in the middle of the production day! Let's say we now write 10 records to file A. If we then have a program B that deletes one record and completes (closes the file), CPF4653 will not be sent as only 10 percent (1 of 10) of the records in file A are deleted. If we rerun program B, again deleting one record and closing the file, we will again not see the CPF4653 as the message is sent when the file exceeds the percentage of deleted records. If we now run program B a third time, we will see the CPF4653 message in QSYSOPR as we now have 30 percent of the records in file A deleted. At this point, we introduce program C or even a system command such as Display Physical File Member (DSPPFM), which opens file A for input only. When program C closes file A, we will continue to see CPF4653 being logged to QSYSOPR and QHST as we still exceed the DLTPCT value for file A. Now, let's have program D write one record to file A and close the file. We will continue to see the CPF4653 message being logged as the current percentage of deleted records is 3/11 or roughly 27 percent. As we continue to call program D, writing one record to file A and then closing the file, we will continue to see the CPF4653 message for the twelfth, thirteenth, and fourteenth records. However, on the fifth call to program D (writing the fifteenth record), message CPF4653 will no longer be logged on the close as we now have gone below the DLTPCT threshold (3/15 = 20 percent). Now we reach an application decision point. Do we want to reorganize file A because it was beyond the DLTPCT threshold earlier, or do we want to not reorganize file A due to the current percentage of deleted records being below the threshold? Either approach could be right for a given file, so in the sample exit program you will see one approach of how to handle this situation either way. You will also have noticed that the CPF4653 message is logged multiple times for file A. Besides making sure that the system operator won't scream because of all of these messages, you will also want to ensure that the application program that schedules file reorganizations reorganizes file A only once. We certainly would not want six messages for file A to generate six consecutive runs of RGZPFM!

To handle these considerations, we will create a control file with one record per library/file/member combination that might be reorganized. At a minimum, one composite key for this file will be by library name, file name, and member name. In the control file record format, we will have the field REORG to indicate whether or not a CPF4653 message has been received for the file and a separate field DLTPCTSTS to control whether the current percentage or historical percentage of deleted records controls the reorganization. The watch exit program will maintain this file. A second program, Purge Deleted Records (PRGDLTRCDS), which is typically run in the middle of the night, will use this control file to determine those file members to reorganize. The name of the control file is RGZPFMLST for Reorganize Physical File Member List.

Though not covered by the sample program, additional control fields would most likely be called for in a production application. Here are some that come to mind:

• ORDER--The Reorganize Physical File Member (RGZPFM) command can reorganize a file while maintaining original record sequence, reorganize a file based on the physical file access path, or reorganize a file based on a logical file access path. The preferred organization would be worth knowing when performing the RGZPFM.
• TIMEST (Time Estimate)--Some files may take longer to reorganize than we have free time for on a given night. Even though the DLTPCT threshold has been exceeded, you may want a control field that contains an estimate for the time needed to reorganize a given physical file member. If this time estimate is less than the remaining time or window for the current run, reorganize the file. If the time estimate exceeds the remaining time, then defer the RGZPFM to another day.
• WCHTIM (Watch Time)--The "deferral" policy associated with the field TIMEST may then suggest another control field: a record of when the first CPF4653 was sent since the last RGZPFM. By using a composite key of REORG and WCHTIM file member reorganizations that had been deferred from an early day, you could then get priority processing on subsequent days. This prioritization could be accomplished, for instance, by having the scheduling program read the underlying control file sequentially by key (the full key being REORG concatenated with WCHTIM), using a partial key of REORG set to 'Y'. This prioritization would hopefully provide more opportunity for the estimated time that the RGZPFM requires to fall into the available time window.
• RMVLFS (Remove Logical Files)--Some RGZPFMs can run much faster if all logical files over the physical file are first removed, then the RGZPFM run, and the logical file members then added back. The scheduling program could conditionally perform this additional processing based on a 'Y' value for this field.

But these types of considerations (and others) are more application-oriented than API-oriented, and this is after all "The API Corner." I will however point out that implementing a feature such as RMVLFS will be much easier if you are aware of the List Database Relations (QDBLDBR) API documented here. If you are unfamiliar with list API processing, then you will want to review my previous articles on the topic. One discussion of list processing starts with the article "Module, Module, Who's Got My Module?" And with all of the information about a message that is provided to a watch exit program, it should be no surprise that the time a watch message is sent (WCHTIM from the previous list of possible control fields) is already available to you. It's field ESCMT00 of the ESCQWFM data structure. The field ESCMT00 is in an internal format that we won't get into at this time, but the value of the field is such that you can use it "as is" in the proposed composite key.

With that background, here is the source for the RGZPFMLST control file:

                R RECORD                           

                  FILE          10                

                  LIBRARY       10                

                  MEMBER        10                

                  REORG          1                

                  DLTPCTSTS      1                 

                K LIBRARY                         

                K FILE                            

                K MEMBER                          

To create the physical file RGZPFMLST into library VINING you would use this command:

CRTPF FILE(VINING/RGZPFMLST) SRCFILE(QDDSSRC)

Within RGZPFMLST, the field REORG will be set to the value 'Y' if the member needs to be reorganized. A 'Y' value essentially means that at least one CPF4653 message has been sent since the last RGZPFM of the member. A value of 'N' indicates that there have been no CPF4653 messages sent for this member. This value of 'N' will be set by the RGZPFM scheduler, PRGDLTRCDS, after the member has been successfully reorganized.

The field DLTPCTSTS indicates whether the scheduling program should determine if the current number of deleted records causes the DLTPCT threshold to be exceeded (a value of 'C' for current) or if any historical exceeding of the DLTPCT threshold is sufficient to do the RGZPFM (a value of 'H' for historic). The default value for this field is 'H' when the exit program WCHCPF4653 writes new records to RGZPFMLST. A user could, however, change this value from 'H' to 'C', in the appropriate records of RGZPFMLST, using utilities such as DFU, SQL, etc. This user-made change would then be preserved by the WCHCPF4653 exit program and the related PRGDLTRCDS scheduling program.

Here is the source for the exit program WCHCPF4653:

fRgzPFMLst uf a e           k disk                                   

dWchCPF4653       pr                  extpgm('WCHCPF4653')           

d Type                          10    const                          

d SsnID                         10    const                           

d Error                         10                                   

d MsgDta                              likeds(ESCQWFM)                

dWchCPF4653       pi                                                 

d Type                          10    const                          

d SsnID                         10    const                          

d Error                         10                                    

d MsgDta                              likeds(ESCQWFM)                

 /copy qsysinc/qrpglesrc,escwcht                                     

dCPF4653          ds                  based(RplDtaPtr)      

d                                     qualified             

d Member                        10                          

d File                          10                          

d Library                       10                          

dRplDtaPtr        s               *                         

dYes              c                   'Y'                   

dHistorical       c                   'H'                   

dRqdNbrParms      c                   4                      

dSessionName      c                   'F_DLTPCT  '          

 /free                                                       

  monitor;           

  // Check to make sure program environment is as expected          

  if %parms < RqdNbrParms;                                          

     dsply ('WCHCPF4653 received only ' +%char(%parms) + ' parms'); 

     *inlr = *on;                                                   

     return;                                                        

  endif;                                                             

  if Type '*MSGID';                                              

     dsply ('WCHCPF4653 received type ' + Type);                    

     Error = '*ERROR';                                              

     *inlr = *on;                                                   

     return;                                                        

  endif;                                                             

  if SsnID SessionName;                                          

     dsply ('WCHCPF4653 received SsnID ' + SsnID);                  

     Error = '*ERROR';                                               

     *inlr = *on;                                                  

     return;                                                       

  endif;                                                           

  if MsgDta.ESCMID00 'CPF4653';                                 

     dsply ('WCHCPF4653 received message ' + MsgDta.ESCMID00);     

     Error = '*ERROR';                                             

     *inlr = *on;                                                  

     return;                                                       

  endif;                                                           

  // Everything looks OK so get message replacement data           

  RplDtaPtr = %addr(MsgDta) + MsgDta.ESCORD;                       

  // Reflect this member in the RGZPFM list                           

  chain(e) (CPF4653.Library :CPF4653.File :CPF4653.Member) RgzPFMLst; 

  select;                                                             

     when %error;       

          dsply ('WCHCPF4653: Environment error 1: ' +   

                 %char(%status));                                                                       

     when not %found(RgzPFMLst);                                      

          Library = CPF4653.Library;                                   

          File = CPF4653.File;                                        

          Member = CPF4653.Member;                                    

          Reorg = Yes;                                                

          DltPctSts = Historical;                                      

          write Record;                                               

     when %found(RgzPFMLst);                                          

          if Reorg = Yes;                                             

             unlock RgzPFMLst;                                        

          else;                                                       

             Reorg = Yes;                                         

             update Record;                                        

          endif;                                                  

     other;                                                       

          dsply 'WCHCPF4653: Environment error 2';                

  endsl;                                                           

  // Set normal end of processing so we continue to be called     

  Error = *blanks;                                   

  return;           

  on-error;                

     Error = '*ERROR';     

     *inlr = *on;          

     return;               

  endmon;                  

 /end-free          

Assuming that the target library you used when creating the RGZPFMLST file is in your current library list, you can create the watch exit program WCHCPF4653 into library VINING using this command:

CRTBNDRPG PGM(VINING/WCHCPF4653)

To start a watch session named F_DLTPCT (File DLTPCT watch), specifying WCHCPF4653 as the exit program, for message CPF4653 being sent to QHST you would use this:

STRWCH SSNID(F_DLTPCT) WCHPGM(VINING/WCHCPF4653) WCHMSG((CPF4653))

WCHMSGQ((*HSTLOG))

To end the session, you would use the ENDWCH command:

ENDWCH SSNID(F_DLTPCT)

The non-error paths of the exit program should contain no surprises for you. The program verifies the environment it is being called in and then makes the appropriate changes to the RGZPFMLST record, which corresponds to the library/file/member identified by the CPF4653 message sent to QHST. The RGZPFMLST file is opened implicitly on the first call the system makes to WCHCPF4653 and will remain open with normal returns from WCHCPF4653 as the program does not set on *INLR except for error conditions.

If a record for the library/file/member is not found, WCHCPF4653 writes a record after setting REORG to 'Y' and DLTPCTSTS to the default value of 'H'. If a record for the library/file/member was found, WCHCPF4653 examines the retrieved value for REORG to determine if the member is already marked for reorganization. If so, WCHCPF4653 simply releases the lock on the record as the file was opened and read for update. If not, WCHCPF4653 changes the REORG value to 'Y' and updates the record.

If an error is returned when trying to chain to the RGZPFMLST file, WCHCPF4653 displays the message 'WCHCPF4653: Environment error 1' along with the %status value for the error. This situation most often indicates that the record is currently locked for update by another job on the system. This error should never happen as there are only two programs that access RGZPFMLST (WCHCPF4653 and PRGDLTRCDS), and these programs are designed such that WCHCPF4653 will never attempt to update a RGZPFMLST record currently being updated by PRGDLTRCDS. This error strongly suggests an environmental error--most likely either that someone has added a third program that uses the RGZPFMLST file in a manner that conflicts with the design assumptions of WCHCPF4653 and PRGDLTRCDS or that someone has modified the PRGDLTRCDS program in a way that again conflicts with the original design. In a future article, we will spend more time looking at the design assumptions that are being used in our two programs. In the event that this error does occur, WCHCPF4653 does not end the watch session. It reports that an error was encountered but continues to watch for other CPF4653 messages that may be sent on the system. This is done as there is nothing in the error situation that suggests future CPF4653 instances will encounter the same problem (assuming a %status value of 1218--record is currently locked).

Another error of the same type, meaning that it should never happen, is the dsply on the other operation of the select group. As the select group has already checked for both a record found and a record not found condition, it is difficult to imagine how the other operation would be in effect unless the WCHCPF4653 program has been modified. If the help desk ever gets a call indicating that this message has been displayed, then the support personnel will have a good idea of where to start looking in the program-a much better situation than having the program continue processing without providing any external indication that the unexpected has occurred.

It was previously pointed out that the file RGZPFMLST remains open across calls to the exit program. This is great from a performance point of view, but when does the file close? The answer, given how the program is currently coded, is that the file is implicitly closed when the session is ended. This works, but what if the exit program wanted to perform some cleanup activity before ending (in addition to having open files closed)? In V5R4, there is no direct way to have the exit program know when it is being ended. I have, however, successfully used an indirect approach. Rather than having the exit program only watch for the application-related message (CPF4653 in the current scenario), the exit program also watches for an additional message, which is control-related. This additional message would be user-defined with perhaps a message ID of END9001 and a 10-character replacement variable. The replacement variable would contain the name of the watch session to end. The current watch session F_DLTPCT would then watch for both message CPF4653 with no comparison data and for message END9001 with comparison data of F_DLTPCT. When the END9001 message is sent, the watch exit program would get control and then perform any necessary housecleaning activity.

In V6R1, watch support has been enhanced such that the exit program can be made aware of when the session is to end. So the indirect approach described above is necessary only if you are on V5R4. And in case you're wondering, the V6R1 enhancement also provides for a special call to the exit program when the session first starts so that any first-time initialization can also be easily accomplished. This initialization support isn't as critical, though, as the exit program could have always used an initialization variable to control first-time processing requirements.

The V6R1 enhancement is a new parameter for the STRWCH command. The new parameter--Call watch program, or keyword CALLWCHPGM--controls when the watch exit program is called by the system. The system always calls the exit program if a watched-for event (message) is sent, but you can specify additional times to call the exit program with these special values:

• *WCHEVT--The exit program is called only for the watched-for event. This is the default value.
• *STRWCH--The exit program is also called when the watch session is starting.
• *ENDWCH--The exit program is also called when the watch session is ending. This ending could be the result of an ENDWCH command, an ENDJOB command, etc. The watch exit program will not, however, be called if an error within a previous call to the exit program is causing the watch session to be ended.

In the next two articles, we will look at the program PRGDLTRCDS, which reads the RGZPFMLST control file and reorganizes the physical file members. As that time, we will also have the opportunity to look briefly at the Retrieve Member Description (QUSRMBRD) API as it is used when supporting a DLTPCTSTS value of 'C', the value that indicates PRGDLTRCDS should only reorganize the member if the current percentage of deleted records is greater than the files DLTPCT value.

Meanwhile, if you have other API questions, send them to me at This email address is being protected from spambots. You need JavaScript enabled to view it.. I'll see what I can do about answering your burning questions in future columns.

Bruce Vining is president and co-founder of Bruce Vining Services, LLC, a firm providing contract programming and consulting services to the System i community. He began his career in 1979 as an IBM Systems Engineer in St. Louis, Missouri, and then transferred to Rochester, Minnesota, in 1985, where he continues to reside. From 1992 until leaving IBM in 2007, Bruce was a member of the System Design Control Group responsible for OS/400 and i5/OS areas such as System APIs, Globalization, and Software Serviceability. He is also the designer of Control Language for Files (CLF).A frequent speaker and writer, Bruce can be reached at This email address is being protected from spambots. You need JavaScript enabled to view it.. 

---

MC Press books written by Bruce Vining available now on the MC Press Bookstore.

		IBM System i APIs at Work Leverage the power of APIs with this definitive resource. List Price $89.95 Now On Sale

Also Read

The API Corner: Moving Jobs Within and Across Job Queues

The API Corner: What Time Zone Do You Want That Date and Time In?

The API Corner: How Do You Want That Date and Time?