Well, I'm finally back from my stay at Club Med! I want to thank Don Denoncourt for taking over the duties of "The Linux Letter" during my absence. I really enjoy Don's writing style. He has a way of cutting directly to the heart of a subject and can stitch together a fascinating article on virtually any topic. Thanks, Don!

One of the frequently asked questions in iSeries news groups (and the focus of many iSeries tech tips) is how to automate FTP transfers. This month, we'll look at a handy tool that will solve the problem for FTP as well as most other interactive programs. It's available not only for my favorite operating system but also for that other popular desktop operating system. Let's jump right in and get better acquainted with this month's subject, the expect program.

The Problem

If you haven't seen the solution to the FTP scripting problem before, you may be surprised by its simplicity. Just fire up your favorite text editor and enter into a text file the commands that you'd issue to FTP interactively. Once you have saved the file, start FTP and use redirection to send the contents of the file to FTP as a string of commands. The following example illustrates an FTP session where I logged onto Red Hat's site and retrieved the README file for Version 8.0 of the Linux distribution. The text appearing in boldface type is what I keyed interactively as input to FTP. The -n parameter disables automatic login in the FTP client.

ftp -n
ftp> open ftp.redhat.com
Trying 66.187.224.51...
Connected to ftp.redhat.com (66.187.224.51).
220 Red Hat FTP server ready. All transfers are logged. (FTP)
ftp> user anonymous This email address is being protected from spambots. You need JavaScript enabled to view it.
230 Login successful. Have fun.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd /pub/redhat/linux/8.0/en/doc
250 Directory successfully changed.
ftp> get README
local: README remote: README
227 Entering Passive Mode (66,187,224,51,209,4)
150 Opening BINARY mode data connection for README (442 bytes).
226 File send OK.
442 bytes received in 0.00581 secs (74 Kbytes/sec)
ftp> quit
221 Goodbye.

If we extracted all of the commands I entered and placed them into a text file, which I'll call ftp.txt, we would get a file containing the following lines:

Now, we can get the README file any time we want by issuing this command:

This method is simple and works well...as long as nothing goes wrong. But what if the file you want has been moved to another directory? What if the server is down? The problem with this simple method is that there's no mechanism for error handling or recovery. If you were typing these commands interactively, you'd know that there was trouble in paradise the moment you tried to get the file. But in a scripted environment, you may not notice. If you require a more robust and reliable solution, then you'll need to take a different tack.

The Solution

You have two options in solving this conundrum--hire someone to babysit the application or implement some software babysitter to do it for you. Obviously, the best solution is to use software. My choice is a program called expect.

Expect does its magic by spawning your desired program, redirecting stdin and stdout to itself. Thus, it can issue commands to the program and monitor the program's responses. The result is the equivalent of a virtual human running the program!

Let's revisit our README retrieval case study and, this time, rewrite it as an expect script, as shown below:

As you can see, there isn't much to this script. The first line is a *nix notation that indicates to the command processor which program will process the script. Normally, you invoke the script by typing expect ftp.txt, which passes "ftp.txt" to expect as a parameter. Using this notation, you simply can type ./ftp.txt and *nix will do the right thing. In either case, anything that follows a pound sign (#) on a line is considered a comment.

The second group of statements initializes variables (accessible from the script by preceding the variable with a dollar sign in the same fashion as the BASH shell). Since we're all good programmers here, I don't need to discuss the importance of never embedding constants throughout a program.

Next, the FTP program is spawned. Following a "send" command, expect will send the text to ftp's stdin and wait for the client program to send the "expected" text back. The "r" shown in each command is translated by expect into a return character, as if the user pressed the return key.

As you can see, the basics are simple. A basic script is comprised of send/expect statement pairs. Lather, rinse, and repeat for each pair of send/expect commands required to accomplish your desired task.

The only error handling I did in this simple script was to ensure that expect would timeout if the FTP client didn't receive the expected response (or any response) from the server within two minutes. Naturally, a more sophisticated script would take additional action.

Are You Expecting?

As so many of those infomercials say, you probably want to know how to get expect and how much it costs. Well, like almost everything I mention, the expect program is free. How you obtain expect depends upon which operating system or distribution you're using. Those who use the Red Hat Linux distribution can easily obtain expect and all of its package dependencies via the up2date utility by issuing the command: up2date -i expect. I believe that Debian users have a similar option through the use of apt-get, the Debian equivalent of up2date. Users of other Linux distributions, and Windows users, can obtain the software from the Expect home page.

Unfortunately, this program comes with scant documentation. However, plenty of sample scripts are provided, and for simple projects, that's all you'll need. For those who are really enthused about Expect's possibilities, you'll find that there is a book available titled Exploring Expect (O'Reilly, ISBN 1-56592-090-2). According to the Expect Web site, this is the definitive guide to using this program.
I haven't even begun to explore the power of expect. The rather simplistic example I use here pales in comparison to the incredible sophistication demonstrated in its sample scripts. I expect that, once you get started, you'll find lots of uses for this wonderful tool. Give it a try!

Barry L. Kline is a consultant and has been developing software on various DEC and IBM midrange platforms for over 20 years. Barry discovered Linux back in the days when it was necessary to download diskette images and source code from the Internet. Since then, he has installed Linux on hundreds of machines, where it functions as servers and workstations in iSeries and Windows networks. He recently co-authored the book Understanding Web Hosting on Linux with Don Denoncourt. Barry can be reached at This email address is being protected from spambots. You need JavaScript enabled to view it..