Simon Coulter leaves us a gift before he departs.

As you probably know, Simon Coulter, a knowledgeable and versatile IBM i expert, passed away in October 2010. Simon was loved by many IBM i developers because he was one of the giants in the industry, on whose shoulders others could stand and reach for new heights.

Among all his posts in the news lists at midrange.com, there was one in which Simon introduced a technique that is critical to analyzing MI instruction streams of OPM programs: Re: QPROCT and QPRODT. In this post, he pointed out that the description information for MI instructions, in form of a 17-byte entry for each instruction, is stored in a hidden space object, QSYS/QPROCT. Additionally, he discovered the format of the 17-byte entries through experiments. This is the key to analyzing the MI instruction stream of OPM programs. Starting from this point, we finally have the opportunity to see clearly the format of MI instruction streams of OPM programs, which has never been publicly documented by IBM. Furthermore, by combining the instruction stream analysis technique with IBM's documentation on other components of OPM programs—such as the object definition table (ODT)—you can create an MI disassembler for OPM programs.

The Mystery Uncovered by Simon

Simon pointed out that the space object QSYS/QPROCT (with object type code/sub-type code hex 19DA) stores 17-byte entries, one for each MI instruction, beginning from offset hex 466 in its associated space. The number of those entries varies from release to release. By observing the dump result of QPROCT, you can find out the number is 230 for V5R4 and 468 for V5R2. Simon also provided the format of the 17-byte entries for MI instructions.

The above format of 17-byte entries for MI instruction in QPROCT revealed a couple of critical keys for analyzing the instruction stream:

• The instruction name/operation code (op-code) mapping, which allows you determine the instruction name by a parsed op-code in the instruction stream
• The number of required operands for a specific instruction

Appendix A at the end of this article shows a complete list of instruction name/op-code mappings (along with numbers of required parameters) that was extracted from the QPROCT at release V5R4.

With Simon's help, it has become possible to clarify the format of an OPM program's instruction stream.

Analyzing an OPM Program's Instruction Stream, Step by Step

As we know, it's possible to retrieve the MI instruction stream component of an OPM program with proper observability via the MATPG (Materialize Program) instruction. The MATPG instruction can also return the following types of OPM program information that are necessary to an MI disassembler:

• The object definition table (ODT)—The ODT contains the views of all objects referred to in the instruction stream other than those objects that are immediate value operands in the instructions. An instruction operand in the instruction stream that contains a reference (possibly qualified) to an object in the ODT is called an "ODT object reference." The ODT consists of ODT directory vector (ODV) entries and optional ODT entry string (OES) entries. An ODV entry is required for each ODT object reference in the instruction stream.
• The HLL breakpoint offset mapping (BOM) table—The BOM table stores the mappings from breakpoints in an OPM MI program to corresponding MI instruction numbers. A breakpoint in an MI program is produced by a break (BRK) directive statement in the MI source of the program. The BOM table component can be used by compilers to relate HLL statement numbers to instruction numbers.
• The HLL symbol table that relates HLL names to ODT numbers
• The object mapping table (OMT)—The OMT entries correspond one for one with the ODV entries; each OMT entry gives a location mapping for the object defined by its associated ODV entry.

According to IBM's documentation on MATPG, you can get the following information about the materialized instruction stream:

• The instruction stream component consists of a 4-byte binary value that defines the total length of the instruction stream component and a variable-length array of 2-byte entries that defines the instruction stream.
• The 2-byte entries define instruction operation codes, instruction operation code extenders, or instruction operands.

Actually, the format of the 2-byte entries belonging to a single MI instruction in the instruction stream might optionally have one to four parts, like the following. Entries in brackets are optional.

The Operation Code Entry

The first 2-byte entry for an instruction in the instruction stream is the op-code entry of the corresponding MI instruction. Note that an op-code entry extracted from the instruction stream cannot be directly used to match an op-code value in the QPROCT; only some of the bits of the op-code entry should be used to represent the real op-code of the MI instruction.

After extracting the real op-code, you can map the result op-code to the name of the instruction via the instruction name/op-code mappings stored in the QPROCT.

The Operation Code-Extender Entry

For MI instructions that use the branch form (B form) or the indicator form (I form), at least one resultant condition must be specified. All resultant conditions are categorized into four groups, and conditions in the same group are represented by the same value in an op-code-extender entry in the instruction stream. There can be up to four resultant conditions specified for a single MI instruction. If an instruction uses B form or I form (the bit 3 of byte 0 of its op-code entry is 1 and the bit 4 of byte 0 is 1), then the following facts are true:

• The 2-byte entry after the op-code entry is an op-code-extender entry that indicates the resultant conditions being specified with this instruction.
• Each 4 bits of the 2-byte op-code-extender entry represent a resultant condition specified with the instruction. The values for resultant conditions in different groups are showed in the following table.
• After the 2-byte entries representing the required instruction operands of the instruction, there are 2-byte entries in the same order for each branch target operand or indicator target operand corresponding to each resultant condition in the op-code-extender entry.

The following table shows the values of all 4-bit op-code extenders. For the resultant condition keywords belonging to each group, see Appendix B at the end of this article. Note that by adding N to the beginning of a resultant condition keyword, you can form a not condition. For example, the code for "not equal" is NEQ.

The Instruction Operand Entries

In the instruction stream of an instruction, after the op-code entry and an optional op-code-extender entry, there are zero to four 2-byte entries that represent the required operands of the instruction. The number of required operands of an instruction is available via the QPROCT. Note that if an instruction uses short form (S form), the number of instruction operands in the instruction stream will be one less than the number of required operands.

According to IBM's documentation on Program Object Specification, there are three types of instruction operands:

Null operands—The value of a null operand in the instruction stream is hex 0000.

Immediate operands—The value of an immediate operand is encoded in the 2-byte instruction operand entry directly. Immediate operands may have the following values:

• Signed binary representing a binary value of negative 4,096 to positive 4,095.

• Unsigned binary representing a binary value of 0 to 8,191.

• Byte string representing a single byte value from hex 00 to hex FF.

• Absolute instruction number representing an instruction number in the range of 1 to 8,191.

• Relative instruction number representing a displacement of an instruction relative to the instruction in which the operand occurs. This operand value may identify an instruction displacement of negative 4,096 to positive 4,095.

ODT object references—This type of operand contains a reference (possibly qualified) to an object in the ODT. Operands that are ODT object references may be simple operands or compound operands.

• The value encoded in the operand refers to a specific object defined in the ODT. Simple operands consist of a single 2-byte operand entry.

• A compound operand consists of a primary (2-byte) entry and a series of one to three secondary (2-byte) entries. A compound operand may be a subscript reference, a substring reference, or an explicit base reference.

The format of immediate operands is as follows:

For example, the value of an unsigned binary scalar immediate operand 8191 is hex 3FFF; the value of a signed binary scalar immediate operand -55 is hex BFC9; and the value of a 1-byte character immediate operand 'A' is hex 20C1.

The format of the primary operand of a compound operand is shown in the following table:

The secondary entries of a compound operand can be either an immediate operand or a simple operand, but cannot be another compound operand. For a subscript reference, after the primary entry, there is one secondary entry that specifies the index value to an element of the array. A substring reference has two secondary entries; the first specifies the value of substring start position, and the second specifies the value of substring length. An explicit base reference has a secondary entry that specifies the pointer on which to base the object for this operand. A subscript reference to an explicit base reference is the combination of a subscript reference and an explicit base reference; the secondary entries of it are an operand that specifies the pointer on which to base the object for this operand followed by one that specifies the index value to an element of the array. A substring reference to an explicit base has three secondary entries: an operand that specifies the pointer on which to base the object for this operand, an operand that specifies the value of the substring start position, and an operand that specifies the value of the substring length.

The Branch Target or Indicator Target Operands Entries

For MI instructions that use B form or I form, at least one resultant condition must be specified for an instruction using B form or I form. After the 2-byte entries representing the required instruction operands of the instruction, there are 2-byte entries specified in the same order for each branch target or indicator target operand corresponding to each resultant condition in the op-code-extender entry.

Put It All Together

Now let's exploit the techniques discussed above in a practical example. Assume that you have a compiled OPM RPG program called HELLO and you already have the parsed ODT table of HELLO at hand. The source of HELLO is like the following (HELLO.RPG).

The beginning 4 bytes of HELLO's instruction stream component indicates that the total size of the instruction stream is hex 00000646 (1606 bytes). For simplicity, we will run through only a piece of the whole instruction stream, which corresponds to the 7th line in HELLO's source code. The ODT object references being used in this piece of instruction stream are listed in the following table:

The part of instruction stream corresponding to HELLO's seventh line is highlighted in the following table:

Assume the start instruction number is hex 2F. Let's apply the above-mentioned analyzing techniques to each instruction in this piece of instruction stream.

Finally, let's compose the disassembled MI instructions and labels represented by branch points (.LKF1000, .LKF2000, and .LKF3000) into the final MI source code corresponding to HELLO's line 7, "C ARR,2 LOKUPARR,WHR 91".

Notes on the disassembled MI source:

• Instruction 2F—Set off indicator *IN91 by copying the value '0' into it via CPYBLA.
• Instruction 30—Copy the value of packed(2,0) scalar WHR into an internal bin(4) scalar .I.
• Instruction 31—Compare array element ARR(.I) with ARR(2) and conditionally branch to label .LKF2000 if ARR(.I) is larger than ARR(2).
• Instruction 32—Increment the value of array index .I by 1.
• Instruction 33—Loop if array index .I is not larger than constant 3.
• Instructions 34, 35—After the loop, if there is no array element that fulfills the search criteria, copy the value 1 into WHR and then branch to the instruction hex 38 that is labeled by branch point .LKF3000.
• Instruction 36—When an array element fulfills the search criteria and the control flow is branched from instruction 31 to instruction 36, set on indicator *IN91 by copying the value '1' into it.
• Instruction 37—Save the current array-index value back to packed scalar WHR.

Congratulations! You've just managed to disassemble the instruction stream corresponding to one line of OPM RPG source statement. If you are new to MI, you might be a little surprised by the amount of work needed to implement an RPG LOKUP (Look up) operation at the MI level.

We can now look through the instruction stream of an OPM MI program. This is owing to Simon's work on the QPROCT object. Rest in peace, Simon.

Appendix A: Instruction Name/Op-Code Mappings (V5R4)

According to the format of the 17-byte QPROCT entries given by Simon, we can extract the op-code, name, and number of required operands for each MI instruction from the space object QSYS/QPROCT. The following 230 records are extracted from a QPROCT object at release V5R4.

Appendix B: Resultant Condition Keywords

as/400, os/400, iseries, system i, i5/os, ibm i, power systems, 6.1, 7.1, V7, V6R1