Say what you will about the AS/400 Division and its often conservative ways, it has always been the most eager adopter and developer of industry-standard benchmarks for gauging the performance of midrange servers. IBM is out there in front of the pack again, trying to drum up support for its new Business Object Benchmark for Java benchmark test. The test, nicknamed jBOB, looks like it could be as useful in gauging the horsepower of servers running enterprise Java applications as RAMP-C and TPC-C, two popular midrange tests, have been on past midrange server families.

The AS/400 Division of cross-platform benchmarking was on the cutting edge in 1989, using IBM’s internal Requirements Approach to Measuring Performance-COBOL (RAMP-C) debit/credit benchmark test to pit the newly announced Silverlake AS/400s against DEC VAXen, IBM mainframe-style 9370 and 4381 minicomputers, and Hewlett- Packard HP 3000 minis. RAMP-C was one of the first cross-platform benchmarks because it was written in COBOL, a language that most minicomputers have supported since having been commercialized in the mid-1970s. The furor that IBM’s audited-yet-unsanctioned RAMP-C tests caused in the midrange in general and among DEC and HP in particular led directly to the establishment of the Transaction Processing Performance Council (TPC), a consortium of server vendors, several years later. The TPC controls the standards and auditing of the various TPC benchmark tests with which we are all familiar. While the RAMP-C test and the industry-standard TPC-A and TPC-B debit/credit online transaction processing (OLTP) tests were all fairly simple, TPC-C, the current online test, was important because its workload was similar to those running on AS/400s in the real world. Moreover, auditors of the TPC test required stringent data and transaction protection software to be implemented on the machines in the TPC-C test, which meant that vendors pushing UNIX or NT equipment could not take shortcuts.

Who’s on First?

Not surprisingly, the AS/400 Division was the first computer vendor to publish results for the TPC-C benchmark with the AS/400 E series of machines back in 1992, and since that time, the TPC-C test has become the best and most widely used OLTP benchmark in the history of the computer business. Part of the reason for its popularity is that the world’s top server vendors all have a say in how TPC-C (and, indeed, other TPC tests) have evolved over time to reflect changes in computer hardware and software over the past seven years. But getting those vendors to play friendly is no small task, and perhaps that is why IBM

has taken the bull by the horns once again and modified the TPC-C test to create jBOB instead of waiting around for the TPC consortium to get its act together on Java.

The TPC and the other main independent benchmarking organization, the Standard Performance Evaluation Corporation (SPEC), have other fish to fry and seem content to let IBM take the lead on enterprise Java benchmarks. SPEC is concentrating on an improved Web-serving benchmark called SPECweb99. The TPC is a year late getting a revised TPCC test out the door and probably won’t get it done for another year, because of bickering and infighting among the consortium. The TPC is also trying to get vendors together on a new e-business benchmark called the TPC-W, and it is unclear how that is going.

Suffice it to say, neither group is focused on developing server-side Java benchmarks. SPEC does, however, have a client-side Java benchmark called JVM98, but it is not appropriate for measuring midrange servers running Java applications. And even the widely used VolanoMark benchmark, developed by Volano, a chat software vendor, is not exactly a good way to measure Java application performance on servers, because it measures the performance of chat software, not the OLTP workloads that are common at AS/400 and other midrange server sites. By the way, nothing prevents companies from using Java code in the forthcoming TPC-W e-business test or in the current TPC-C OLTP test, but because Java performance—even on compiled Java bytecodes—is still generally a fraction of the performance of native C++ or RPG, vendors will probably stick with their current highly tuned native coding rather than change to Java.

But just because vendors don’t or won’t use Java on the TPC-C test doesn’t mean that the basic architecture of the test is not suitable for creating a Java benchmark. IBM’s Commercial Processing Workload (CPW) test, for instance, is also based on the TPC-C benchmark; IBM replaced RAMP-C with CPW three years ago as the main measure of relative performance of AS/400 machines. So it seems obvious that IBM would base its jBOB test on the TPC-C test. However, jBOB is not, says IBM, simply CPW reimplemented in Java.

Tweaks Make Comparisons Tough

There are some major differences between jBOB, CPW, and TPC-C, so anyone making comparisons has to be careful. The TPC-C test simulates the OLTP associated with managing a distribution warehouse. There are five different transactions in the TPC-C suite: new order, payment, order status, delivery, and stock level. The TPC-C performance metric is expressed in transactions per minute, and it counts only the number of new orders that a server can process. Generally, the new orders represent about 45 percent of the transaction throughput regardless of machine type. The TPC-C test has variable end user key and think times, and it is generally implemented in a three-tier architecture (meaning the database and application are on separate machines) rather than a two-tier setup (where the database and applications are on the same machine, which is how most of the real world does things). TPC-C is not a set of code that vendors run on their machines to test them but a specification that they must adhere to as they develop their own TPC-C applications.

The CPW test is a modified TPC-C application suite implemented in RPG rather than COBOL. The CPW metric is a new order count on a two-tier setup with just enough mathematical shenanigans (it looks like an unofficial TPC-C new order count divided by lucky number 7) to keep the TPC-C (which requires that all vendor-announced TPC-C results be audited and given price/performance ratings) from getting mad.

The jBOB test can be implemented in either a two- or three-tier configuration, but IBM has initially done it on the AS/400 in a two-tier setup, just as it did with CPW. The jBOB test counts the total number of transactions on the TPC-C suite (not just new order transactions), which is written in 100% Pure Java that has been fully optimized as it is crunched down into Java bytecodes. The jBOB test uses Java Database Connectivity (JDBC) to access the AS/400’s DB2/400 database rather than the native RPG or COBOL access methods used in the CPW and TPC-C tests. The jBOB test has fixed (rather than

variable) key and think times. Most significantly, jBOB is not a specification but a set of Java code that IBM and its benchmarking partners will distribute in bytecode (not source code) form to companies that want to run the jBOB test. IBM says that the Java in the jBOB test reflects the current state of Java programming in commercial settings and that it doesn’t want computer vendors—including other IBM divisions and, specifically, Microsoft—to start playing games with the jBOB applications to tweak its Java to make use of server-specific or operating system-specific advantages. The idea with jBOB is to stick with Java’s “write once, run anywhere” philosophy, and IBM correctly sees that the only way to maintain that is to strictly control jBOB’s source code. Of course, no one in the industry will stand for this, so don’t be surprised if jBOB eventually gets spun out into a separate organization. (The TPC is not interested at the moment.)

jBOB’s Your Uncle

The results of the jBOB tests, which IBM ran on S30 and 730 series AS/400s running OS/400 V4R3 and V4R4, are presented in a white paper that you can get at www. as400.ibm.com/whpapr/ jbob400.htm. These results show that an AS/400 730-2067 with four 262-MHz Northstar processors, 6 GB of main memory, and 276 GB of disk capacity and running OS/400 V4R4 was able to process 103.8 jBOB transactions per second (JTPS) running at 98 percent of total CPU capacity with a total of 2,240 concurrent jBOB users. An S30-2259 server with four 125-MHz Apache processors, 4 GB of main memory, and 105 GB of disk capacity and running OS/400 V4R3 was able to process only 41 JTPS at 99 percent of CPU with a total of 900 users. That’s a 2.5-times performance improvement between the Apache and Northstar generations. While much of that improvement comes from doubling the processing power and L2 cache memory in the AS/400’s chips by moving from Apache to Northstar processors, a respectable portion comes from V4R4’s integration of the 1.1.7 release of Sun Microsystems’ Java Virtual Machine (JVM) and much-improved JDBC drivers.

While IBM’s white paper doesn’t mention it, the techies in Rochester have run the jBOB test on other platforms supporting other American National Standards Institute (ANSI)-compliant SQL databases. While developing the jBOB test, IBM matched up a four-way Compaq ProLiant server running Windows NT 4.0 against a four-way AS/400 Apache server. That Compaq machine used Pentium Pro processors, which are a little thin on L2 cache memory—only 1 MB of L2 cache compared to 4 MB on the Apache chip—but the Pentium Pros ran at a higher 200-MHz clock speed, so the L2 cache advantage was offset somewhat. (Tests like TPC-C and CPW are geared toward stressing a server’s memory and I/O bandwidth, while Java tests like jBOB are sensitive to clock cycles and L2 cache memory sizes.) IBM’s NT-AS/400 bakeoff on the jBOB test on this equipment showed that the NT server more or less kept pace with the Apache AS/400 server until just under 700 jBOB users were added to each system. At that point, the jBOB transaction rate on the NT server started decreasing as the number of jBOB users increased. The Apache server, mostly because of its larger L2 cache, was able to keep adding users until it peaked at 900 jBOB users. (See Figure 1 for details.)

Now, those old Pentium Pro chips in the ProLiants were no great shakes compared to the 550-MHz Pentium III Xeon chips that Intel is now shipping and that have 2 MB L2 cache memories that run at the same clock speed as the Pentium III processor as well as main memory that runs at 100 MHz rather than 66 MHz. But servers using these fast Pentium III Xeons will probably still lag behind IBM’s latest AS/400 Northstars. Just as was the case with the Apache-Pentium Pro four-way server comparison, the Northstars have half the clock speed but four times the L2 cache memory of the Pentium III Xeons. Both OS/400 and NT 4.0 can be equipped with the latest 1.1.7 JVM. Next year, when Intel jacks up clock speeds in the 32-bit IA-32 processor line and starts shipping 64-bit IA-64 Merced processors in the 1 GHz range, IBM’s AS/400 Division will have I-Star processors running at 560 MHz in February and 800 MHz by the end of 2000. And the AS/400 will

have more scalability, with 24-way symmetric multiprocessing (SMP) configurations compared to eight-way Intel-based PC servers using the new Profusion chipset that was announced in June, and will start shipping by the end of summer. The AS/400 Division seems to have an impressive lead over Wintel servers. The trick—and one that IBM has failed to perform time and again with the AS/400—is to make sure the AS/400 keeps that lead and leverages it.

The Java Penalty: Hope of Parole

What is on the minds of most AS/400 users is the question of when Java will become a practical language in which to develop their own applications. IBM likes to tell customers that the performance penalty in using Java is about 2 to 1 right now. While this is probably true for Java code that is getting native database access through IBM’s VisualAge and Java toolkits rather than JDBC access (which is essentially Java-ized SQL and hence very slow), working backwards from the TPC-C and jBOB tests makes it look as if the penalty for the Java-JDBC combo were more like 5 to 1 compared to RPG applications and native database access. (I can hear IBM and the TPC fuming that I would even make such a comparison, but someone has to.) But extrapolating jBOB end-user and transaction rates from results on the Apache and Northstar servers to future I-Star, Power4, and Power5 servers (a dubious practice given how Java will change between now and 2004, when the Power5 chips will start shipping), it looks as though next year’s 24-way I-Stars will be able to support more Java OLTP users than a 12-way Apache server could support on native RPG workloads. The follow-on I-Stars due at the end of 2000 will have about as much Java OLTP power as the 12-way Northstars had in RPG OLTP power.

When the Power4 processors ship, the gap between RPG and OLTP performance will shrink dramatically because the Power4s (as well as the follow-on Power5s) will have dual-math processors that are each twice as powerful as the single-math units in the Apache, Northstar, and I-Star family. These three AS/400 chips do one floating point instruction per cycle, but the Power4 will have two whole microprocessors on a single piece of silicon, with each micro-SMP able to process eight instructions per cycle. None of these math units does much in the way of helping RPG performance, but they make all the difference in the world to Java, which does floating point, not decimal, math. By the time the Power4s actually ship in early 2001, improvements in Java coding, JVM performance, and JDBC access and the added math power in the Power4 chip will probably close the Java performance gap to 2 to 1 and may even reach parity if we are lucky. If it doesn’t work out that way, IBM may be tempted to pull its old “golden screwdriver” trick to artificially cripple RPG performance on I-Star, Power4, or Power5 servers and make Java application speeds equal to that of geared-down RPG, thereby limiting one marketing problem (and creating another).

The point is that the Northstars and the Java support in OS/400 V4R4 are only just the beginning. Within a year or so, Java will be an affordable alternative to RPG because IBM will really start cranking up the hardware underneath Java. Presumably, this hardware will also be phenomenal at running RPG applications, which is important as customers shift from RPG to Java. One way to think about it is that the lower cost of supporting today’s RPG applications on future hardware will somewhat offset the higher costs of supporting tomorrow’s Java applications (which presumably will be easier to develop once the pool of talent is available) on those same future AS/400s. Java will run on plain vanilla servers, too, and not need to be enhanced with costly interactive features. It seems that the AS/400 and its customers will be ready for enterprise Java applications at just about the same time. Now, all IBM has to do is price it right, and customers will follow its lead.