Discover the essence and usage of the COL associated with a library.

In the reply post in 2009 to a question called "Best Way to Check if a Library Has At Least 1 Changed Object" in the midrange-l mailing list, CRPence mentioned the term "changed object list" (COL). The following is the original reply post by CRPence. (Sorry, Chuck, I don't know your full name.)

Subject: Re: Best Way to Check if a Library Has At Least 1 Changed Object

From: CRPence <crpbottle@xxxxxxxxx>

Date: Wed, 25 Nov 2009 17:37:59 -0800

List-archive: <http://archive.midrange.com/midrange-l>

> Richard Schoen wrote:

>

>     I'm putting together a process to back up changed libraries.

>     What's the fastest way to identify if at least one object in a library

>     has changed within the last 24hrs so I can back it up.

>

>     One option is SAVCHGOBJ, but I want to spin through al list of all

>     libraries and identify any where at least 1 object has changed.

Although others have suggested DSPOBJD and that might seem nice, consider that members are not objects [manifest to the user nor DSPOBJD] at the library level. Although the first data change should be reflected in the *FILE, to be sure all database members were included, I would also prefer to use the DSPFD to get either *MBR or *MBRLIST output for its change information. Regardless...

What is really desired here is the "changed object list" for the library. I described something about an actual usage of the COL in the APAR text for SE17466 which can be found at:

http://www-01.ibm.com/support/docview.wss?uid=nas3b826e7ab1b60ac3b86256f420052ae9c

The MATCTX instruction provides the ability to provide selection for objects modified since a specified modification date\timestamp. I infer the "programming note" can be handled best, by passing the last full save date for the library. This would likely be just what the SAVCHGOBJ does to generate its list of objects. Unlike DSPOBJD, the MATCTX does return internal object types like the object type *MEM [aka x/0D50].

http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/rzatk/MATCTX.htm

Regards, Chuck

The "changed object list for a library"? Sound a little strange? Yes, this term occurs in the IBM i Information Center only once (in documentation of the Materialize Context (MATCTX) MI instruction). Let's take a trip to visit it!

Background Information: Basic Structure of a Library (a Context MI Object)

At the MI level, a library (*LIB) is a context MI object with MI object type code hex 04. In the rest of this article, the terms "library" and "context" will be used interchangeably. The MI object type code/subtype code of a permanent context object is hex 0401, while that of a QTEMP library of each job (MI process) is hex 04C1. The basic structure of a context object is like the following.

Figure 1: This is the basic structure of a context MI object (version 1). (Click images to enlarge.)

• Each MI object starts with a 256-byte MI object header in which common MI object attributes are stored. So does a context object.
• A context-specific header in which context specific attributes are stored occurs after the MI object header.
• A machine index exists in a context object immediately after the context-specific header, each entry of which contains information necessary to address an MI object residing in the context by symbolic object ID (aka MI object type and MI object name). Details of a context machine index entry are available in section "Index Entry Format shared by COL Machine Indexes and Context Machine Indexes." In the remaining part of this article I will refer to this machine index as context machine index.
• Each context object has a space MI object (with object type code/subtype code hex 1952 and external object type *OIRS) called the Object Information Repository (OIR) space associated with it. The OIR space (OIRS) associated with a library stores object description information (e.g., extended attribute, user-defined attribute, and text description of the object) for external objects residing in the library. Each OIRS entry consists of one or multiple 512-byte blocks.
• Each context object has an independent index MI object (with object type code/subtype code hex 0E90 and external object type *QDIDX) associated with it. This index is designed to improve the efficiency of locating the object description information of an external object in a library, in other words locating the OIRS entry of an external object in the library's associated OIRS. For each OIRS entry, there is an index entry in this index containing the following information of the corresponding OIRS entry: 2-byte MI object type code, 10-byte external object name, bin(4) OIRS entry length (in unit of 512 bytes), bin(4) OIDRS entry offset (in unit of 512 bytes). The beginning 12 bytes (MI object type code and external object name) of an index entry is the key portion.
• System pointers to the OIR index and the OIR space are stored in a context object's associated space.

Find Out the COL!

Before we get started, I'd like to clarify a concept: which operations to an MI object change the modification time of it?

• Does an update operation to the content of a space object change the space object's modification time? No.
• Does an enqueue operation to a User Queue object change the queue object's modification time? Yes.

The accurate answer to this question is that it's up to the system to determine which operations to which type of MI objects should change the modification time of an MI object of that type. Here is a detailed explanation on this question: How the server updates changed object information with the SAVCHGOBJ command.

To achieve our goal of this section, we will need a really sharp tool, the System Service Tools (SST). As mentioned above, each MI object has a 256-byte MI object header (at the start of the base segment of it) that stores object attributes that are common to all types of MI objects. After the 256-byte MI object header is the object-specific header in which attributes unique to the specific type of MI object are stored. Examining the hexadecimal content of a context's base segment in SST, you can find that a context object has a 256-byte object-specific header after its MI object header, from offset hex 0100 to offset hex 0200 in its base segment. The following is the first 512-bytes (MI object header and context-specific header) of a context object called LSC dumped via SST.

<PRE>

      DISPLAY/ALTER/DUMP                                                                            11/10/10  10:18:30   PAGE      1

 CONTEXT                      SUBTYPE:  01      NAME:  LSC                                 ADDRESS:  3D77C7A06D 000000

 BASE SEGMENT

    3D77C7A06D 000000  DISK UNIT  0007 DISK PAGE  000FCA00

    3D77C7A06D 000000    00010030039000B4  3D77C7A06D000000   

0001000000000000  2505AC72B8001000  *..........G~_...................*

    3D77C7A06D 000020    81000401D3E2C340  4040404040404040   

4040404040404040  4040404040404040  *a...LSC                         *

    3D77C7A06D 000040    4040800000002000  000000783F100601   

95110076B06F4000  27DCBEA8DC000000  *  ..............n...^? ....y....*

    3D77C7A06D 000060    0000000000000000  000000000D000000   

3D77C7A06D000100  4002000000000000  *..................G~_... .......*

    3D77C7A06D 000080    951DF990A562C000  0000000000000000   

0000000000000000  0000000000000000  *n.9.v.{.........................*

    3D77C7A06D 0000A0    0000FF1C0000002B  0000000000000000   

0000000000000000  0000000000000000  *................................*

    3D77C7A06D 0000C0    0000000000000000  E000000000000000   

0000000000000000  0000000000000000  *........$.......................*

    3D77C7A06D 0000E0    0000000000000000  0000000000000000   

002B000000000000  0000000000000000  *................................*

    3D77C7A06D 000100    3D77C7A06D000200  0000002B00000000   

0000000000000000  0000000000000000  *..G~_...........................*

    3D77C7A06D 000120    0000000000000000  0000000000000000   

0000000000000000  0000000000000000  *................................*

                         2 LINES    3D77C7A06D 000140    TO    3D77C7A06D

000160    SAME AS ABOVE

    3D77C7A06D 000180    0C93914503000200  951DF98FBA02E000   

8000080800000000  C000010B873EFF80  *.lj.....n.9.[.$.........{...g...*

    3D77C7A06D 0001A0    0000000000000000  0000000000000000   

0000000000000000  0000000000000000  *................................*

                         2 LINES    3D77C7A06D 0001C0    TO    3D77C7A06D

0001E0    SAME AS ABOVE

    3D77C7A06D 000200    00000000EAEA0401  D4C1C3C8C9D5C4E7   

0000000000000028  0000000000000006  *........MACHINDX................*

</PRE>

To get the content of a context object in SST, you may need to type the STRSST command and then select the following menu items.

<PRE>

1. Start a service tool

   4. Display/Alter/Dump

      1. Display/Alter storage or 2. Dump to printer  

         1. Machine Interface (MI) object  

            4. Permanent context (04)         

               1. Find by object name 

                  1. Hexadecimal, alter capable    

</PRE>

Pay attention to the two 8-byte data items at offset hex 0180 and hex 0188! Yes, that's what we are looking for:

• 8-byte Single Level Store (SLS) virtual address of the COL at offset hex 0180 (hex 0C93914503000200 in the above dump output)
• 8-byte timestamp value (in the form of a system clock) of current COL date/time at offset hex 0188 (hex 951DF98FBA02E000 in the above dump output)

By simple experimentations, you can find out that the current COL date/time of a library is updated to the save date/time each time the library is saved—for example, as a result of a SAVLIB operation. To get the readable format of the timestamp value associate with the COL shown above, you may call an example RPG program provided by i5toolkit, t173.rpgle like the following:

<PRE>

> CALL T173 x'951DF98FBA02E000'

  DSPLY  SAA timestamp    2011-10-09-17.16.02.894894

</PRE>

Next, to find out what the COL is exactly, let's examine the content addressed by the COL's SLS virtual address. Type the STRSST command and then select the following menu items:

<PRE>

1. Start a service tool

   4. Display/Alter/Dump

      1. Display/Alter storage or 2. Dump to printer  

         5. Starting address

</PRE>

Then, enter the COL's SLS virtual address. The output might be like the following:

<PRE>

      DISPLAY/ALTER/DUMP       

11/10/11  17:32:51   PAGE      1

 DUMP ITEM

    0C93914503 000000  DISK UNIT  0005 DISK PAGE  000DACC2

    0C93914503 000200    00000000EAEA0401  D4C1C3C8C9D5C4E7   

0000000000000037  0000000000000001  *........MACHINDX................*

    0C93914503 000220    0000000000000006  000000000000002B   

0000000000000000  0000100000000000  *................................*

    0C93914503 000240    0880000000000000  901B000000000000    

0000000000000000  0000000000000000  *................................*

    0C93914503 000260    0000000000000000  0000000000000000   

0000000000000000  0000000000000000  *................................*

    0C93914503 000280    0C93914503001000  0000000000000001   

8800000000000000  0000000000000001  *.lj.............h...............*

</PRE>

Wow, seems so familiar! Is it a machine index? The answer is yes, and additionally, we can walk through the COL machine index entries, which represent the changed objects in a library via SST. Start SST and select the following menu items:

<PRE>

1. Start a service tool

   4. Display/Alter/Dump

      1. Display/Alter storage

         2. Licensed Internal Code (LIC) data

            8. Machine index

</PRE>

You will be led to the Run Index Operation panel. To locate the lowest entry of the COL index, enter the COL's 8-byte address into the Base Page field, set the Option field and the Function field to 1 and FINDLT (Find lowest entry), respectively, and then press ENTER. To walk through all the other COL index entries, use Option=3 plus Function=FINDNX and press Enter repeatedly. The following is a screenshot of the Run Index Operation panel.

Figure 2: The example COL entry is displayed in the Run Index Operation panel.

Note that the Result area displays the target index entry returned, and the Argument area allows you to specify the search argument data.

The Same Index Entry Format Shared by COL Machine Indexes and Context Machine Indexes

By investigating the index entries in a COL, you can find out that a COL shares the same index entry format with that of a context machine index. The variable-length index entry format consists of the following fields:

• 2-byte MI object type/subtype code
• Variable-length MI object name in which whitespaces (hex 40) are compressed. The following rules apply to compressing whitespaces in the original 30-byte MI object name. A non-whitespace character remains unchanged. One or more white spaces between two non-whitespace characters are converted to: ' ' + -N (where -N means the one-byte signed binary representation of the negative value of number of whitespaces). For example, nine contiguous whitespaces between two non-whitespace characters are converted to hex 40F7. If there are remaining whitespaces after the last non-whitespace character, they are converted to ' ' + N (where N means one-byte binary representation of the value of number of whitespaces). For example, 19 whitespaces is converted to hex 4013.Therefore, the name of an MI object called 'A         A                   ' is converted to hex C140F7C14013, while the name of an MI object called 'CCCDDDEEE CCCDDDEEE           ' is converted to hex C3C3C3C4C4C4C5C5C540FFC3C3C3C4C4C4C5C5C5400B.
• 6-byte SLS virtual address of the MI object

The above index entry format reveals that an IBM i library manages the addressability of MI objects residing in it by maintaining a mapping from symbolic object ID to SLS virtual address via the context machine index. Also, the addressability of all changed objects in a library is maintained in the same manner via the COL machine index. Finally, the addressability of all libraries is maintained in the same manner in a special context object, the machine context. To examine MI objects stored in the machine context index, follow the steps mentioned in the previous section with the Base Page field set to the address of your machine context index's base page (for example, hex 000000000D000200 at V5R4 or previous VRMs).

The Conclusion on COL

Now we've seen that the Changed Object List (COL) associated with a library is a machine index with the same entry format as a library's context machine index. The COL date/time is a reference timestamp that records the date/time of the last save operation on the library. The COL is designed to improve the performance of materializing (and addressing) changed objects in a library. I infer that the system provides the following functionality to system utilities or user programs that need to know which objects were changed in a library via the COL and the COL date/time of the library:

• The requester requests a COL with a specified modification date/time.
• System compares the given modification date/time with the COL date/time of the library. If the given modification date/time is less than the COL date/time of the library, all entries in the library's context machine index will be tested. If the given modification date/time is greater than or equal to the COL date/time of the library, only the objects in the library's COL index will be tested. If the target library is a very big one and is saved regularly, this condition will certainly improve the performance of the requested operation dramatically—for example, a SAVCHGOBJ operation or the example program shown at end of this article, which materializes all changed objects in a library since the library was last saved.

Now, it's time to improve the image "Basic Structure of a Context MI Object (Version 1)" by adding what we've just discovered, the COL and current COL time. The following is the improved version.

Figure 3: This is now the basic structure of a context MI object (version 2).

Work with COL Programmatically

The only information we can acquire about the COL via all existing programming interfaces (APIs and MI instructions) available to user programs is the 16-byte extended materialization template of the MATCTX MI instruction. (The page introducing MATCTX is also the only place where the term "changed object list" appears throughout the piles of documentations on IBM i.) The 16-byte extended template provides the following critical information about the COL associated with a context object:

• Whether a COL is associated with a context object
• Whether the COL is usable
• The date/time currently associated with the COL

Note that all context objects except QTEMP are in the system domain and hence cannot be accessed via the MATCTX instruction from a user state program at security level 40 or above. Fortunately, IBM provided a system state wrapper for MATCTX, the QusMaterializeContext API that is exported from service program QSYS/QUSMIAPI.

An example RPG program provided by i5toolkit, t170.rpgle, demonstrates the steps to work with COL information in the extended template of MATCTX. T170 accepts a 10-byte library name as its only parameter, tests whether the specified library has an associated COL, and optionally displays the date/time associated with the library's COL. By calling T170, you may find out that most IBM-shipped libraries and all user-created libraries do have associated COL. The following libraries do not have associated COL: QSYS, QRECOVERY, QSRV, and QTEMP.

A Working Example of Checking Whether a Library Has Changed Objects

At the end of the article, I'm glad to attach a working example RPG program as an answer to the question mentioned at the beginning of this article: Best Way to Check if a Library Has At Least 1 Changed Object. Certainly, the prerequisite is that the target library has been saved at least once. Steps of the example program are quite straightforward:

1. Retrieve the target library's COL date/time by materializing the library's extended context attributes via the MATCTX instruction.
2. Materialize all context entries in the target library by modification time, setting the timestamp field in MATCTX's materialization option template to the library's COL date/time.

If the target library is saved regularly, this program will run very fast, since only the objects in the COL are checked.

The following is the example RPG program, t174.rpgle, provided by i5toolkit.

<PRE>

     /**

      * This file is part of i5/OS Programmer's Toolkit.

      *

      * Copyright (C) 2010, 2011  Junlei Li.

      *

      * i5/OS Programmer's Toolkit is free software: you can redistribute it and/or modify

      * it under the terms of the GNU General Public License as published by

      * the Free Software Foundation, either version 3 of the License, or

      * (at your option) any later version.

      *

      * i5/OS Programmer's Toolkit is distributed in the hope that it will be useful,

      * but WITHOUT ANY WARRANTY; without even the implied warranty of

      * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

      * GNU General Public License for more details.

      *

      * You should have received a copy of the GNU General Public License

      * along with i5/OS Programmer's Toolkit.  If not, see <http://www.gnu.org/licenses/>.

      */

     /**

      * @file t174.rpgle

      *

      * List changed object list of a library after the last full save.

      */

     h dftactgrp(*no) bnddir('QC2LE')

     fQSYSPRT   o    f  132        disk

      /copy mih-comp

      /copy mih-ptr

      /copy mih-pgmexec

      /copy mih-ctx

      /copy mih-undoc

     d i_main          pr                  extpgm('T174')

     d   p_ctx_name                  10a

     d prt_chg_obj     pr

     d   bytes                       10i 0

     d   mod_ts                       8a

     d rtmpl           ds                  likeds(rslvsp_tmpl)

     d mopt            ds                  likeds(matctx_option_t)

     d mtmpl           ds                  likeds(

     d                                       matctx_receiver_ext_t)

     d                 ds

     d ctx                             *

     d ctx2                            *   overlay(ctx)

     d                                     procptr

     d col_time        s             26a

     d num_chg         s             10i 0

      * output fields

     d obj_num         s              5s 0

     d ws              s              1a

     d obj_type        s             25a

     d obj_name        s             30a

     d i_main          pi

     d   p_ctx_name                  10a

      /free

           if p_ctx_name = 'QTEMP';

               ctx = qtempptr();

           else;

               // resolve system pointer to target context object

               rtmpl = *allx'00';

               rtmpl.obj_type = x'0401';

               rtmpl.obj_name = p_ctx_name;

               rslvsp2 (ctx : rtmpl);

           endif;

           // retrieve current time of the COL

           mopt = *allx'00';

           setbts(%addr(mopt.sel_flag) : 4);

             // materialize extended context attributes

           mtmpl = *allx'00';

           mtmpl.base.bytes_in = %size(matctx_receiver_ext_t);

           QusMaterializeContext(mtmpl : ctx2 : mopt);

           if tstbts(%addr(mtmpl.ext_attr) : 0) = 0

              or

              tstbts(%addr(mtmpl.ext_attr) : 1) > 0;

                 dsply 'No usable COL' '' p_ctx_name;

                 *inlr = *on;

                 return;

           endif;

           // retrieve changed context entries

           mopt = *allx'00';

           setbts(%addr(mopt.sel_flag) : 7); // retrieve symbolic ID

           setbts(%addr(mopt.sel_criteria) : 3);

             // select by modification date/time

           mopt.timestamp = mtmpl.col_time;

             // use current COL time as search criteria

           mtmpl = *allx'00';

           mtmpl.base.bytes_in = %size(matctx_receiver_t);

           QusMaterializeContext(mtmpl : ctx2 : mopt);

           num_chg = (mtmpl.base.bytes_out - %size(matctx_receiver_t))

             / %size(sym_object_id_t); // (bytes_out - 96) / 32

           if num_chg > 0;

               dsply 'Number of changed objects' '' num_chg;

           else;

               dsply 'No changed objects in' '' p_ctx_name;

               *inlr = *on;

               return;

           endif;

           prt_chg_obj(mtmpl.base.bytes_out : mopt.timestamp);

           *inlr = *on;

      /end-free

     oQSYSPRT   e            CHGOBJREC

     o                       obj_num

     o                       ws

     o                       obj_type

     o                       ws

     o                       obj_name

     p prt_chg_obj     b

      /copy apiec

     d qlicvttp        pr                  extpgm('QLICVTTP')

     d   cvt_type                    10a   options(*varsize)

     d   ext_type                    10a

     d   mi_type                      2a

     d   ec                                likeds(qusec_t)

     d cvthc           pr                  extproc('cvthc')

     d                                1a   options(*varsize)

     d                                1a   options(*varsize)

     d                               10i 0 value

     d mopt            ds                  likeds(matctx_option_t)

     d mtmpl           ds                  likeds(matctx_receiver_t)

     d                                     based(tmpl_ptr)

     d tmpl_ptr        s               *

     d oid             ds                  likeds(sym_object_id_t)

     d                                     based(pos)

     d pos             s               *

     d i               s             10i 0

     d cvt_type        s             10a   inz('*HEXTOSYM')

     d mi_type         s              4a

     d ec              ds                  likeds(qusec_t)

     d prt_chg_obj     pi

     d   bytes                       10i 0

     d   mod_ts                       8a

      /free

           tmpl_ptr = %alloc(bytes);

           // retrieve changed context entries

           mopt = *allx'00';

           setbts(%addr(mopt.sel_flag) : 7); // retrieve symbolic ID

           setbts(%addr(mopt.sel_criteria) : 3);

             // select by modification date/time

           mopt.timestamp = mod_ts;

             // use current COL time as search criteria

           mtmpl = *allx'00';

           mtmpl.bytes_in = bytes;

           QusMaterializeContext(mtmpl : ctx2 : mopt);

           num_chg = (mtmpl.bytes_out - %size(matctx_receiver_t))

             / %size(sym_object_id_t); // (bytes_out - 96) / 32

           pos = tmpl_ptr + %size(matctx_receiver_t);

           ec.bytes_in = %size(ec);

           for i = 1 to num_chg;

               obj_num = i;

               // convert returned MI object type code to external objec type name

               obj_type = *blanks;

               qlicvttp(cvt_type : obj_type : oid.type : ec);

               // in case QLICVTTP failed on internal obj types, e.g. hex 0D50 (*MEM)

               if ec.bytes_out > 0;

                   cvthc(mi_type : oid.type : 4);

                   obj_type = 'MI Type Code: hex ' + mi_type;

               endif;

               obj_name = oid.name;

               except CHGOBJREC;

               pos += %size(sym_object_id_t); // 32

           endfor;

           dealloc tmpl_ptr;

      /end-free

     p                 e

</PRE>

Suppose that there is a newly saved library called LSD. Move the physical file called CCCDDDEEE into LSD and then call T74 like this: CALL T174 LSD. The output of T174 might be like the following:

<PRE>

00001 MI Type Code: hex 0D50    CCCDDDEEE CCCDDDEEE

00002 *FILE                     CCCDDDEEE

</PRE>

An Error in QusMaterializeContext's Documentation

By the way, there is an error in QusMaterializeContext's documentation in the IBM i Information Center, http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/qmatctx.htm

(or its counterparts at version V6R1 or V7R1).

It says: "The Materialize Context (QusMaterializeContext) API returns either the type and subtype of the object or system pointers, based on what you specify for the materialize options parameter. The API returns the information for all or for a selected set of objects that are contained by the context. This information is returned to a receiver variable. If the context is null, the machine context (the QSYS library) is returned."

Pay attention to the last sentence. The machine context is just the machine context that contains the addressability of all other context objects along with several other types of objects (e.g., user profile objects). And the QSYS library is just the QSYS library, a context object whose addressability is stored in the machine context like other libraries. The machine context and the QSYS library are two completely different context objects.

Thank You, Chuck!

Many thanks, Chuck! Thank you for your nice posts at midrange.com. I've learned a lot from you.

Junlei Li is a programmer from Tianjin, China, with 10 years of experience in software design and programming. Junlei Li began programming under i5/OS (formerly known as AS/400, iSeries) in late 2005. He is familiar with most programming languages available on i5/OS—from special-purpose languages such as OPM/ILE RPG to CL to general-purpose languages such as C, C++, Java; from strong-typed languages to script languages such as QShell and REXX. One of his favorite programming languages on i5/OS is machine interface (MI) instructions, through which one can discover some of the internal behaviors of i5/OS and some of the highlights of i5/OS in terms of operating system design.

 

Junlei Li's Web site is http://i5toolkit.sourceforge.net/, where his open-source project i5/OS Programmer's Toolkit (https://sourceforge.net/projects/i5toolkit/) is documented.

Also Read

Programming in ILE RPG - Defining Data Structures, Part I

H and F Position-less Specs

Practical RPG: AI and IBM i