Composite Keys and Key Lists

So how do you deal with composite keys? Do you use a work field, and move all the pieces in? Or are you somewhat more advanced, using a data structure? Whatever technique you used in the past, you have to learn a new method to work with composite keys in externally described files.

Judging by the queries on the Midrange Computing BBS, and in discussions with newly minted AS/400 programmers, it seems that the area of composite keys, key lists, and the rules for using key lists with RPG operations is confusing. The rules are really very simple, and the flexibility gained by converting to external files and using the new features is well worth the problems you may initially have.

Just What Is a Composite Key?

If you are coming from the System/34 to the AS/400, you may be baffled at the idea of composite, noncontiguous key fields. Even System/36 programmers may not feel that they are on solid ground, since the "alternate index" features arrived relatively late in the game. Also, many System/36 uses of alternate indexes are rather simple. An example is defining an alternate index on the Customer Master over the customer name, rather than the customer number.

The S/36 implementation also imposes some limits on how you define an alternate index, in that you are limited to three noncontiguous fields. That means you have to be careful when designing record formats, so that you can create alternate indexes over all of the fields that you need.

The AS/400, as did the System/38 before it, changed all of the rules, and also made programming quite a bit easier. On those machines, an external file can have any number of fields defined as index fields, as long as they stay within the limit of 120 bytes total. I have never come close to the limit, but I suppose there are some who wish the limit was bigger.

Also, with external files on the database machines, you have many more options for defining each key field. For example, you can define a key field to be in descending, rather than ascending, sequence. This could be useful if you are printing a listing of balances due, or inventory for an 80/20 report. The list of options that you can specify for key fields is quite extensive, and well worth reviewing if you are new to the machine. You may find that programming jobs that used to require tricks or many steps can now be handled as a matter of "definition."

The point is, on the AS/400, you can define different conditions for each field that is part of the key. And that is what a composite key is: two or more fields, from anywhere in the record format, that you want to use to access the file. Please remember that you define keys for external files in the DDS for the file, not on the F-spec in your RPG program. Although the system internally references the key fields in terms of from-to positions, you only have to think of the key fields in terms of the name of the field. Stop worrying about "from" and "to" positions! It's time to let the machine take care of the details that it is concerned with, so that you can deal with things on a higher level.

How Do I Define a Composite Key?

Although defining a composite key is not really within the scope of this article, we will touch upon it briefly. Refer to 1. This depicts the definition of a physical file, with a "primary" key defined within it. Also, a logical file is defined over the physical file, defining an "alternate index," composed of multiple, noncontiguous fields.

Although defining a composite key is not really within the scope of this article, we will touch upon it briefly. Refer to Figure 1. This depicts the definition of a physical file, with a "primary" key defined within it. Also, a logical file is defined over the physical file, defining an "alternate index," composed of multiple, noncontiguous fields.

There is some debate among AS/400 programmers as to the preferred method of defining keys over files. Some say that all keys should be in logical files, and that the physical file should never be keyed. Others maintain that the primary key should be included in the physical file. I have used both methods, and there are reasons to use them both. At this point, I tend to favor the approach of not including keys in the physical file.

Regardless of where you define keys, they are all treated the same within an RPG program. The program must be able to tell the operating system the record that is needed, in terms that the system understands. This means that you must make requests for records accessed by key within the terms that the record was defined. Please read that sentence again, since it summarizes the problems that you will have as you learn how to use keyed external files in your programs.

I Already Know How To Do This!

You may be thinking that you already know how to get at any record you want in a composite key file. If you are a sharp S/36 programmer, you probably create a data structure defining your multiple keys. If you are not quite as experienced, you may use the more laborious MOVEL/MOVE method, of building up little "work fields" until you finally arrive at the key. In either case, these methods won't work with externally-described composite keys, although the data structure people are closer to the new way of doing things.

And just what is this new way? In keeping with the new way of doing things, native RPG includes two new op-codes, KLIST (Key List) and KFLD (Key Fields). Let's look at these op-codes, review how they are used, and figure out why they cause so many problems.

For No Other Purpose

A KLIST is somewhat peculiar. It is used only for accessing a file by key. You use it only on file operations: CHAIN, DELET, READE, REDPE (new), SETGT and SETLL. You can't compare, move, or otherwise use a KLIST. These restrictions underscore the fact that a KLIST is a unique construct within RPG. After all, if you could access a composite key file with the old method of a data structure or a built-up field, then the language would simply let you do that and not need this strange new op-code.

To further add to the strangeness of the KLIST, it can only be coded if it is immediately followed by one or more KFLD statements. And to top it off, the KLIST isn't even executed! I mention this because many programmers innocently include multiple KLISTs in a separate subroutine, and execute that as part of their program initialization. Although this does no harm, it is unnecessary. A KLIST is a "declarative" statement, and is not "executed" in the sense that other RPG op-codes like MOVE and Z-ADD, are executed. Simply including a KLIST anywhere in the source is sufficient; it is "executed," so to speak, when the program is compiled.

Where to place KLISTs is another matter. Some programmers place all KLISTs at the beginning of the program, or within their initialization subroutine. Others place a KLIST immediately before the file access statement that uses the KLIST. I tend to favor the first method, and put all the KLISTs up front. The second method is OK, but breaks down when you use the same KLIST more than once in the program.

A KLIST is named following the rules for field names, even though it can't be used as a field. I start my KLIST names with "@", to distinguish them from other types of fields. You will find it useful to adopt a similar convention for your KLISTs, so that when you see them used, you will know they are KLISTs and not "simple" fields.

What About KFLD?

The other side of the KLIST coin is the KFLD (Key Field), which is really where the misunderstandings creep in. The whole point of a KLIST/KFLD construct is to define a key that you use to retrieve records from a file defined with a composite key access path. The KFLD is the means you use to tell your program to tell the operating system the record that you want.

The rule for defining KFLDs is simple: a KFLD in the same relational position as the composite key field must have the same attributes (character/numeric, length, decimal positions). Please reread that until you understand it. I am astonished at the problems people have with this, since it is a very simple concept. It is practically no work to define a KLIST and the KFLDs that are included with it, since you can work with the listing of the file and copy the definitions right from the listing. For example, refer to 2, which illustrates KLIST/KFLD for the definitions in 1.

The rule for defining KFLDs is simple: a KFLD in the same relational position as the composite key field must have the same attributes (character/numeric, length, decimal positions). Please reread that until you understand it. I am astonished at the problems people have with this, since it is a very simple concept. It is practically no work to define a KLIST and the KFLDs that are included with it, since you can work with the listing of the file and copy the definitions right from the listing. For example, refer to Figure 2, which illustrates KLIST/KFLD for the definitions in Figure 1.

Now, unlike a KLIST name, you can and must move values into the KFLDs before using the KLIST to retrieve records. Apart from the simple restriction of a KFLD not being an array or table name, you can use any field for the KFLD, as long as the attributes are the same as the corresponding field in the file. In practice, I tend to define separate KFLD fields. You can code the length and decimal positions entries on the KFLD statement, so you don't need separate statements to define the KFLDs.

Why define separate fields? Well, I like to have very explicit control over the values in my KLISTs, so I define the fields and move values into them immediately prior to using the KLIST for a file operation. You don't have to use this technique, but be aware that the value used in the KFLD is the "current" value of the field. If you aren't using a field that you explicitly control, the value may not be what you expect.

So Why Can't I Use the Old Way?

At this point, you might wonder why you can't just use the old technique of a data structure or a long field. After all, if a composite key field is 30 bytes long, why not just use a 30 byte field, and move values into the right places?

The problem with that technique is that the operating system demands exact correspondence with key fields, in the order of their appearance. For example, if your 30 byte composite key field comprises fields of length 7, 10, 8, and 5, your first key field must also be length 7. Succeeding key fields must be length 10, 8 and 5. If you try to apply a 30 byte key field to this composite key, you will get horrible compile errors. Your only alternative, of course, is to use the expected technique, which is to define a KLIST made up of KFLDs of the correct attributes.

Now, there's nothing to say that you can't use a data structure to define one of the KFLDs. For example, you might want to break up your 10 byte key field into two or three pieces. There's no problem with that, as long as you use the complete 10 byte key field in the key list. The system doesn't care how you arrive at a value for the key field, it only wants you to use the same attributes as those defined for the corresponding key in the file.

This Seems Like a Lot of Work...

And it is. If there were no greater value than getting at a record, the whole KLIST/KFLD technique would create a massive outcry of dissatisfaction. But in fact, you can use some really neat programming tricks with the KLIST/KFLD structure. The tricks involve using partial keys, which is another ramification of composite keys. Now understand, a "partial key" in this context is not the same as using "part of a key." For example, most of us have had the pleasure of explaining to a user that with our nifty name-search program, they don't have to key in the whole last name, but only the first few characters, and the program will display last names that are in the vicinity of their search request. The programming that you used in that technique is the "part of a key" technique, where you take however much was entered, SETLL with it, and read forward from there.

The partial key technique goes quite a bit further than that, and is extremely useful for a very common data processing task: you can use partial keys to retrieve "sets" of data, with much less programming than might otherwise be necessary.

For example, assume you have an order file; a key might be customer number, order number, and line item number. Using partial keys, you can retrieve all orders for a customer, or all line items for a particular order for a customer. Furthermore, you can define KLIST/KFLD constructs within your program to accommodate these partial keys as shown in 3.

For example, assume you have an order file; a key might be customer number, order number, and line item number. Using partial keys, you can retrieve all orders for a customer, or all line items for a particular order for a customer. Furthermore, you can define KLIST/KFLD constructs within your program to accommodate these partial keys as shown in Figure 3.

A particularly useful op-code for this situation is READE (Read Equal). I was used to the S/38 implementation of this long before the S/36 version became available. I have always felt that the S/36 version was practically useless, because it didn't allow usage of the partial key technique. For example, if the order file is defined with keys of customer, order and order line number, you can use the loop construct shown in 4 to read through all orders for a customer. The indicator on the READE operation is set on when a record is retrieved that is not for the same customer (or at end of file). Compare this with the old RPG II technique of using the simple READ operation and comparing the fields to see if you still are within the same customer.

A particularly useful op-code for this situation is READE (Read Equal). I was used to the S/38 implementation of this long before the S/36 version became available. I have always felt that the S/36 version was practically useless, because it didn't allow usage of the partial key technique. For example, if the order file is defined with keys of customer, order and order line number, you can use the loop construct shown in Figure 4 to read through all orders for a customer. The indicator on the READE operation is set on when a record is retrieved that is not for the same customer (or at end of file). Compare this with the old RPG II technique of using the simple READ operation and comparing the fields to see if you still are within the same customer.

You can extend the technique "from left to right" for as many key fields as you like. For example, 5 shows how you would define and use a construct to retrieve all line items for a particular customer order.

You can extend the technique "from left to right" for as many key fields as you like. For example, Figure 5 shows how you would define and use a construct to retrieve all line items for a particular customer order.

You should review all of the file op-codes that use a key field (KLIST) in factor 1 (CHAIN, DELETE, READE, SETGT, and SETLL). In situations where you use a partial key field, the first record meeting the conditions of that partial key is used. You should pay particular attention to the SETLL and SETGT operations, and also be aware of the special values *LOVAL and *HIVAL.

Watch Out For These

There are some other things you should know about defining KLIST/KFLDs and using composite keys. First, when you use an externally defined file in your program, the compiler prints the key fields immediately after the end of the source. So if you are getting compile errors about key field disagreements, go directly to that part of the listing and compare what you see with what you coded. If the key fields are not at all what you think they should be, then you should go back to the F-spec, and see from what library the compiler pulled in the file definition. If you have files named the same in different libraries, and the key fields are defined differently, then you might be creating problems for yourself when you compile or run the program. You have to be careful, even if the program compiled successfully. When you run the program, the file that is opened must have key fields that agree with what you compiled. Don't try to pull any tricks here, since you won't get very far: you'll get a rather exotic message, about key fields not having the same attributes as expected.

Another area where you'll have to be careful doesn't concern key fields per se, but has to do with the files you use. You have to be aware of the access path rules that are in effect for the file. For example, if the access path excludes certain records, then no amount of fiddling with key fields in your program will retrieve those records. You will simply get "record not found" indications. I mention this because many newcomers spend fruitless hours looking for problems in their programs, and blaming faulty KLISTs, when in fact the problem is completely independent of the program. This is a new concept to S/36 programmers, since alternate index files did not have any select/omit criteria associated with the alternate index. If you get really stuck and think there is a bug with an executing program, use the WRKJOB (Work with Job) command and display the open files and file overrides. You can then refer to the definitions of the open files and determine if the records that you want are in the access path. Remember, if the records aren't in the access path, they aren't in the file, as far as your program is concerned.

A final consideration is what do you do if the access path is made up of only one key field? Should you define a KLIST of a single KFLD, or just use an otherwise available field to access the file? For example, see 6. There is really no great debate favoring either technique, and it will depend upon your programming style. Like a cat at the doorstep, I can't decide either, and have used each technique about half the time. At this point I tend to favor the single KFLD construct, because that way, I am treating all file accesses consistently.

A final consideration is what do you do if the access path is made up of only one key field? Should you define a KLIST of a single KFLD, or just use an otherwise available field to access the file? For example, see Figure 6. There is really no great debate favoring either technique, and it will depend upon your programming style. Like a cat at the doorstep, I can't decide either, and have used each technique about half the time. At this point I tend to favor the single KFLD construct, because that way, I am treating all file accesses consistently.

What You Should Do

If you are just getting started with external files, you should sit down with some listings of your files and make up sample key fields. Pick some of your files that have composite key fields, and sketch out the KLIST/KFLD constructs that you might use. Remember that each KFLD must exactly match the corresponding key field in the file.

If you will study the requirements before getting into coding, you will see that using composite key fields is really a simple concept. But with the simplicity comes programming power that used to involve tedious moves, compares, and other non-intuitive operations. The whole point of RPG/400 is to make your programming life simpler, more powerful, and to the point, and you will find the KLIST/KFLD construct as the most appropriate method of

Unlocking the KLIST

Figure 1 Physical file and logical with composite key

 Physical: A R PFFILE A FIELD1 5 0 A FIELD2 10 A FIELD3 8 A FIELD4 8 A K FIELD1 Logical: A R PFFILE PFILE(PHYFILE) A K FIELD2 A K FIELD4 A K FIELD1 
 
 Unlocking the KLIST
 
 Figure 2 KLISTs for Figure 1
 
 KLISTs for Figure 1 C @KEY KLIST C KFLD @KF1 50 C* C @KEY2 KLIST C KFLD @KF2 10 C KFLD @KF4 8 C KFLD @KF1 
 
 Unlocking the KLIST
 
 Figure 3 Defining partial KLISTs
 
 Defining Partial KLISTs C @ORDKY KLIST C KFLD @CUSNO C KFLD @ORDNO C KFLD @ORDSQ C* C @ORDK1 KLIST C KFLD @CUSNO C KFLD @ORDNO C* C @ORDK2 KLIST C KFLD @CUSNO 
 
 Unlocking the KLIST
 
 Figure 4 Get all orders for a customer
 
 Get All Orders for a Customer C @ORDK2 SETLLORDERS C* C *IN99 DOUEQ'1' C @ORDK2 READEORDERS 99 ----EQ (NE to Key) C *IN99 IFEQ '0' C ... (various statements) 
 
 Unlocking the KLIST
 
 Figure 5 Get all line items for an order
 
 Get All Line Items for an Order C @ORDK1 SETLLORDERS C* C *IN99 DOUEQ'1' C @ORDK1 READEORDERS 99 ----EQ (NE to Key) C *IN99 IFEQ '0' C* ... (various statements) 
 
 Unlocking the KLIST
 
 Figure 6 Only one key field
 
 Only One Key Field C @ORDK2 KLIST C KFLD @CUSNO C* C @ORDK2 CHAINCUSMAS 99 HI---- C* C CUSNO CHAINCUSMAS 99 HI----