The office is closed today, but I am here because I am awaiting the arrival of my IBM CE. During the week, our database server (an IBM pSeries running AIX and Oracle) spontaneously rebooted, rudely cutting off the 30+ users who had been enjoying its services. Once the system was back up (and the phone stopped ringing), we did some forensics and determined that the problem was with the pSeries planar (a.k.a. mainboard, a.k.a. motherboard). Subsequent discussions with IBM Service led to this appointment for a planar replacement--and to my current state of solitary confinement. I don't mind the solitude, though. Without the incessant interruptions of a normal business day, I am able to sit back and reflect on the architecture of the systems that we have recently adopted to run our business. I didn't have to contemplate things for too long before I realized that my musings would provide excellent fodder for a column, which I now share with you.

On the Frontier of n-Tier

The architecture of our new system is n-tier (where n is 2, 3, or 4, depending upon which subsystem you are using). I am sure that most readers are familiar with n-tier design, so I won't go into a detailed discussion in this article. (A quick search on "n-tier architecture" at Google will give you background information if you need it.) For the purposes of this discussion, it is sufficient to know that proponents of n-tier speak glowingly of its scalability, its potential for fault-tolerance, and its ability to utilize the most appropriate component for each "tier." Opponents of n-tier argue that the scalability comes in the form of burdensome complexity, the added complexity begs the need for expensive fault-tolerance, and the most appropriate component is a single machine that does it all--for example, an i5. Which side is right? Both are, as I'll demonstrate shortly.

Let me begin by introducing you to the system--standard n-tier stuff by any metric. Depending upon which way you want to draw your "tiers," you have at the top tier the database server, which is the aforementioned IBM pSeries running AIX Version 5. The database management software running on this box is the also-aforementioned Oracle DBMS. The next tier down (up) is the application server, an IBM xSeries running SuSE Linux Enterprise Server, which hosts a couple of important components: JBoss (the open-source J2EE application server), Apache Tomcat (the open-source Java servlet container) and an instance of the Progress DBMS (a remnant of an earlier version of one subsystem that will eventually be replaced). At the bottom/top of the heap are the Windows XP Professional client machines that run a fat-client Java GUI application, which talks to the application server; a fat-client Windows GUI application, which talks to the application server (calling it a Windows GUI app is redundant, sorry); and the Mozilla Firefox browser, which talks to a couple of Java Servlet applications, also on the application server. If ever there was a melange of software and hardware assembled in one place, this is it!

Just so you don't get the wrong idea, we are running standard, run-of-the-mill business applications. We are not trying to launch nuclear weapons from our desktops. Thus, at first glance, this whole thing may appear to be a system fit to be immortalized in a cartoon strip by Rube Goldberg. In actuality, it does work surprisingly well, but it is not without warts.

The scalability argument used by the proponents of n-tier is why we have separate database and application servers. According the to vendor, companies smaller than ours could get by with a single Linux machine running both functions, but our company's size put us right on the bubble to go either way. Given the incredible bottlenecks that we now experience (how about a billing run that lasts more than 10 hours instead of the 20 minutes that we had on our iSeries-based software?), I'm glad we went with our vendor's recommendation. Actually, I wish I had purchased an even more powerful database server in the first place, as it appears that we'll be throwing more money--er, I mean hardware--at the system in the not-too-distant future to fix those problems. Unfortunately, I had forgotten the first rule of software acquisition: Always add at least 50% more hardware to what your vendor recommends because your sales rep will not want to scare you off by providing realistic hardware requirements in his proposal. (What? Me cynical?)

As for the fault-tolerance argument, I'm told by our vendor that they have built in the functionality to add redundancy all around. I'm not that familiar with Oracle, but I'm sure it must have data-replication features. Furthermore, adding more application servers should be a straightforward task, since all they need do is make connections to a database server. Some elementary tricks with DNS can yield the desired redirections to an active database server, should the need ever arise. In any event, there is no argument that n-tier architecture can't easily be configured for fault tolerance.

Before I delve into the issue of "the right component for the tier," let me make a point that demonstrates the opposing view's argument regarding complexity. As I mentioned at the beginning of this article, I am awaiting the IBM CE to replace the planar of my AIX box, which requires that the box first be gutted, which requires that the box be powered off and unplugged. Unlike the iSeries, which needs a simple PWRDWNSYS command to take down the system down in an orderly manner, the AIX box is running Oracle as an application, not as an integrated part of the OS. So you need to stop Oracle independently of the OS. The vendor has yet to fix my request that the machine do this as part of a shutdown procedure, so I manually issued the command to stop the Oracle instance. Everything was going as planned, but the AIX box seemed to be waiting for something. Then it hit me. The application server had connections to the database server, so the database server would be waiting for those to end before it terminated. "Easily fixed," thought I. Logging onto the application server, I issued the command to shut it down. Eureka! On the database server, I was back at the command prompt and was able to issue the command to shut down. Again, the box sat there waiting. In a minute or so, it began spewing messages about the lack of an NFS connection to the application server.

What we had here was a classic circular dependency, a "chicken-and-egg" situation if you will. In Windows, a network share connection that fails will result in Windows popping up an error message whenever you attempt to use something on that share. In UNIX, the most common network file sharing is done using Network File System (NFS). NFS can act like its Windows counterpart if the connection is made using the "soft" mount option. The default mount option (which they used), however, is to make a "hard" mount. This essentially means that the I/O operation will be retried indefinitely until it succeeds or until you get aggravated enough to yank out the power cord. This condition was where I found myself. Fortunately, the solution was easy enough: Just restart the application server and allow the I/O to complete. Once the DB server's NFS daemon was satisfied, the machine powered down successfully. This demonstrates a simple example of the complexity added by using an n-tier architecture. I'm sure that on a single-host system, this integration is already completed. (At least I'd hope so). I have submitted a bug report for this problem and am awaiting the fix. Until then, I'll have to figure out which services I need to kill on the application server to ensure that I can successfully power off the database server. Just what I need--something else to add to my "to-do" list.

The Best Component for Each Tier

Prior to committing to this software, I flew to the vendor's office for a visit. There, I spoke to the alpha geek responsible for the software design. Frankly, I am quite pleased with most of their tool selections. They make excellent use of the Java open-source tools JBoss and Tomcat. All of the application server components run on the open-source operating system Linux, and all of their development is now in Java. In fact, the Windows GUI components are to be replaced within the next couple of years with Java equivalents. Although they don't officially support it, the components that are already Java-based run just fine on my Linux desktop. During my visit, they told me that they have run it that way in their shop and will eventually fully support their package on a Linux desktop. I assume that the same would be true for MacIntosh OS X users, too.

As we were discussing the various choices that they made, we got around to discussing the choice of the Oracle DBMS. I asked why they chose Oracle over something else, citing DB2 as a good alternative and DB2 on an iSeries as an even better one. My host should be happy that I wasn't drinking coffee when he offered his answer, as he most likely would have been wearing some of it. I nearly choked up with laughter when he replied that "... we chose Oracle over DB2 because DB2 is proprietary...," implying that Oracle wasn't. If anyone can offer an example of something more proprietary than Oracle, then please do so.

Why should I care which DBMS is being used? Had DB2 been chosen instead of Oracle, then with the exclusion of the client software, the entire system could be running on a single i5. Given an option, I would have used a DB2 instance on the OS/400 partition and run all of the services running on the application server in a Linux partition, on the same i5. Sweet!

In addition to reducing my machine count, utilizing this configuration would allow me to simplify the backup and recovery of my software. Currently, I have one tape drive on the database server and another on the application server, with an accompanying array of confusing and non-integrated backup procedures. Had this entire thing been hosted on i5, backup and recovery would be trivial, as it has been on the AS/400 for years. Had this thing been running on a single i5, I could easily create the necessary scripts to start and shut down everything in an orderly fashion, without the issues that I currently have. Had it been put on an i5, we could take advantage of the high-availability options the i5 has had since the days that it was named AS/400. Fault-tolerance would not need to be bolted on; instead, it would be a natural extension of the i5 platform.

There Is Hope

At the time we committed to this software, the i5 didn't yet exist. The machine was still called iSeries, and AIX as an OS partition was not available, so I had no way to run Oracle except as a separate box. (Oracle was available for Linux back then, but it was not available for Linux running on an iSeries partition. Perhaps that will change if it hasn't already.) As you may have guessed, I did try to make this happen!

With all of the improvements that IBM has been making to it, I suspect that the next time I need to update my hardware, I'll be consolidating everything back onto i5. I intend to keep pressuring the vendor to consider supporting this option, since the possibility is tantalizingly close.

What about your project? Is your n-tier product starting to become overly complicated? Perhaps it's time to take a fresh look at your options. By choosing the same open-source tools that IBM has embraced and by reevaluating i5's capabilities, you, too, can tame the n-tier beast and truly enjoy the benefits that architecture offers.

That's it for this month. My CE has arrived, and it's time to roll up the sleeves and tear into the pSeries. If you have any comments or success stories you would like to share, please feel free to email me or to post a message into this article's forum.

Barry L. Kline is a consultant and has been developing software on various DEC and IBM midrange platforms for over 21 years. Barry discovered Linux back in the days when it was necessary to download diskette images and source code from the Internet. Since then, he has installed Linux on hundreds of machines, where it functions as servers and workstations in iSeries and Windows networks. He co-authored the book Understanding Linux Web Hosting with Don Denoncourt. Barry can be reached at This email address is being protected from spambots. You need JavaScript enabled to view it..