In part 1, we processed a directory. In part 2, we process one file in that directory.

Stream files are not database files.

While that statement is obvious to programmers, it's not always clear to the greater community. The end users, the folks whose jobs we are supposed to be supporting, use various forms of stream files to store their data, and they don't understand why we can't for example just "use this spreadsheet" as part of our application. And while that's an interesting philosophical discussion, as programmers we sometimes have to simply get things done, and that in turn means taking whatever data the user sent us. I've spent a lot of time over the years importing data primarily from Excel—and more specifically from comma-delimited files. Two techniques exist: CPYFRMIMPF and parsing the data in RPG. CPYFRMIMPF is a completely different animal that perhaps can be covered another day. Today, I just want to talk about parsing a stream file.

Why DIY?

Why do it yourself? The primary benefit is that you have complete control over the processing. Though convenient, CPYFRMIMPF limits you. As an example, you can't use CPYFRMIMPF if your stream file has varying formats (such as header and detail lines). Or let's say you want to process the file differently based on something in one of the records; with CPYFRMMIPF, you would have to copy all the records to a temporary file and then process that file. That's not necessarily a bad thing; I do exactly that in many of the import subsystems I've written over the years. But let's say you want to use only a few records in a large import file. In that case, you'd have to extract all the records into the temporary file even though you're only using a few.

So today's example is going to focus on reading the stream file yourself. One note: Since this is defined to be a comma-delimited file, I can treat it as a standard human-readable text file. Under other circumstances, you might receive binary data, which requires a completely different access technique. There are actually routines that allow you to directly access an Excel file, as an example. They're in Java and well outside the scope of this article, but I thought you should know.

Processing a Stream File

Stream file contents are accessed using APIs similar to the ones I introduced in the previous article on processing directories. Let's take a look at the prototypes for the APIs I'll be calling in this program.

     dcl-pr IFS_fopen pointer extproc('_C_IFS_fopen');

      file pointer value options(*string);

      mode pointer value options(*string);

     end-pr;

     dcl-pr IFS_fclose extproc('_C_IFS_fclose');

      fd pointer value;

     end-pr;

     dcl-pr IFS_fgets pointer extproc('_C_IFS_fgets');

      pbuf pointer value;

      size uns(10) value;

      fd pointer value;

     end-pr;

They're quite straightforward: open, close, and get. The open and close are very simple. IFS_open takes a file name and an access mode (we'll be using read mode) and returns a file handle. Pass that handle to IFS_close and the file is closed. Of course, opening and closing a file doesn't do you much good without at least one intervening read; that's the job of the get routine IFS_fgets. I also define a few variables.

     dcl-s fp pointer;

     dcl-s sp pointer;

     dcl-s buf varchar(2000);

I use two pointers and a buffer. Note that the buffer is type varchar. I do that so that I don't have to constantly keep track of the length of the data that I've read in, but it does make for a rather strange bit of code later on. But let's start walking through the logic.

       // Open stream file

    fp = IFS_fopen( wFile: 'r');

   if fp = *null;

      SetError('01':%str(strerror(errno)));

      return;

   endif;

This is very simple code. The variable wFile contains a fully qualified file name, and IFS_fopen attempts to open it. The mode is "r", which indicates read-only open. If the file doesn't exist, you'll get an error. The SetError routine is just a placeholder to show you how you might signal an error to the caller. How you handle this sort of fatal error depends very much upon the application. You may also notice the use of %str. When you're using the C APIs, you tend to use that a lot. It converts between traditional RPG character variables and the null-terminated strings that C uses. In this case, the routines strerror and errno combine to return a human-readable error message, but that message is null-terminated. We need %str to use it in our RPG program. Assuming, though, that the open is successful, the variable fp (which has been defined as type pointer) will contain a handle which you can then use for any of the other file-based APIs. And we do that immediately by trying to get the first line.

   // Read in the first line

    sp = IFS_fgets(%addr(buf:*data): %size(buf): fp);

   if sp = *null;

      SetError('02':%str(strerror(errno)));

      return;

   endif;

    buf = %str(sp);

As I said, the IFS_fgets function isn't exactly intuitive, especially to we RPG programmers who haven't traditionally done a lot of pointer-based programming. I have to pass IFS_fgets the address of the buffer and the size of the buffer, along with the file handle that we got from the IFS_fopen API in the previous section of code. If the routine is successful, it returns the same pointer I passed in. If it's not successful, it returns null (that's the test that you see immediately following the API call). But the really confusing part is the next line, where I convert the null-terminated value returned from IFS_fgets into a regular RPG string, which I then place in the variable buf. But you might be asking yourself, why do that? Isn't the data already in buf? Well, yes, it is, but buf is type varchar, and the IFS_gets didn't set the length. Now, you might think I could just set the length of the variable, perhaps by scanning the string for the terminating null:

   %len(buf)= %scan(x'00':buf) - 1;

That looks good, but I found something out through some painful trial and error. It turns out that the compiler really, really wants to be helpful, and if you set the length of a variable, it will fill the variable with blanks. So all that nice data that you just read into the buffer is overwritten with blanks, which certainly defeats the purpose. So I don't do that; instead, I use the previous line, in which I basically overwrite the buffer with itself. It's counter-intuitive, but it works perfectly. So now for the rest of the code.

   // Parse the buffer and process it

    Parse(buf);

   // Close the file

    IFS_fclose(fp);

       return;

And that's all it takes. Now obviously this particular code only processes the first line of the file. That might be enough, especially in an inbound router. The router may read the file, use some logic to identify the transaction type, and then send it on to another program. But this is exactly the scenario where you might need your own parser rather than a generic CPYFRMIMPF.

Where to Go from Here

So clearly this is a very simple application. All of the work will be done in the Parse routine, which I don't even show here. If I get a call for it, I can probably do another article on parsing, but for now this should give you enough of a start on IFS processing to give you some ideas of where it could be helpful in your environment. I've had great success using this to allow users to upload their own data to the system (after running it through validation, of course). Use this basic design to have some fun with the IFS and see what you can come up with.