Is that the right philosophy to follow? Or should you not put off till tomorrow what you can do today? For the answer, think "upgrade."

If it ain't broke, don't fix it.

That old adage has been around since the Carter administration. The down-home advice seems so logical and obvious that everyone may feel compelled to follow it. I know that with my busy schedule, it's easy for me to put off updating software on working systems when there are far more pressing projects to tackle. If you're willing to admit it, you probably can point to a system or two under your control that could use some tender, loving care. Unfortunately, following the "if it ain't broke" philosophy can readily lead to minor irritations and downright spectacular failures. You'd think that 30 years in this business would have provided me with enough opportunities to learn this lesson and that I would have it indelibly imprinted in my mind. But no, I fell into the trap once again.

I have a long-time client who recently contacted me regarding the server I installed in his business three years ago. At that time, the original server I had installed for him had logged 1576 days of uptime—a number he still brags about—but it was getting long in the tooth, causing him some concern. He purchased an IBM xSeries equipped with four 72GB drives, which I configured as a RAID10 array and then loaded with CentOS 5. It does yeoman's duty running Samba, performing as a Windows network primary domain controller (PDC) and file and print server. It also does the basic TCP/IP tasks for which a Linux box serves perfectly: internal DNS and DHCP servers, intranet Web server, Request Tracker server with the associated PostgreSQL database server, and probably a number of things I can't readily recall. Soon after it was placed in service, it inherited additional duties as a VMWare host under which a virtualized external Web and mail server ran. Also "guesting" on the box was an instance of Windows XP whose whole raison d'être was to satisfy a requirement of the QuickBooks software in use by this company.

The purpose for the client's call was to tell me that they were having difficulties using the backup function of QuickBooks. It kept reporting that there was insufficient disk space to perform the backup. I know that they backup to a file share provided by Samba, so I assumed that the error was indicating an error on the backup source. I SSHed into their server and looked. Sure enough, they were down to around of 150 Mb of free space on the file system where that share was stored, and I knew that their QB backup files were larger than that. We discussed the options: expunge the shares of their kruft or add capacity to the server. (There are two free slots for drives in that server.) Given the choice of doing house-cleaning or spending minimal money to increase capacity, he chose what everyone does given those options: spend money! I gave the client a link to a vendor who had NOS IBM drives for this server ($50 each, ye gods!), he placed the order, and we agreed that he would contact me again once the drives arrived to schedule installation.

So the appointed time came, and I arrived at their office on a Saturday to install and configure the new drives. We discussed the benefits and tradeoffs of changing from RAID 10 to RAID 5 and came to the conclusion that to minimize downtime and our time, we'd take the easy path. I'd configure the new pair as RAID 1, which will create what appears as a new drive to the OS, and then add it into the system using logical volume management. I figured that it would take no more than an hour or two to complete the process. Since I was there, I decided that I would take that opportunity to do all of the upgrades that I could since I got to visit the server and client infrequently, and we could never seem to find an acceptable time to take the server down for maintenance and upgrades.

Searching the Web, I discovered that there was a new BIOS upgrade from IBM for the server itself while the ServeRAID card was at the proper level. The server's OS was downlevel, being CentOS 5.5, while the current version is CentOS 5.8. VMWare Server 2.0.2 is the latest and was already installed, so no upgrade was required for it. You may remember VMWare Server 2.0.2 as the last version of VMWare's server product that runs on a host operating system. Subsequent versions require installation on bare metal.

Once I downloaded the necessary ISO images and BIOS update software, I set about doing the actual upgrades. I started by loading the BIOS software so that it was current. Next, I installed the hard drives and booted the server using the ServeRAID management CD. Configuring the two new drives as a RAID volume was trivial, and upon reboot I found things as expected. I used the appropriate tools to mark the new "drive" as a physical volume, added it to the volume group, extended the logical volumes that are used to store /home and the Samba shares, and then extended the file systems. Everything was going as planned, and I found myself ahead of schedule! You can guess where this is going: "Abandon all hope, ye who enter here."

Anyone who uses any of the VMWare products that are designed to be loaded onto an existing operating system, such as the old VMWare Server product and the current VMWare Workstation product, knows that they require a bit of special handling whenever they're started on a new system or one that has had a kernel upgrade. In this case, the special incantation is to run the script "vmware-config.pl" to configure the system to use the new kernel. This is where I entered the IT first circle of hell: "It used to work."

Within a second of issuing that command, I was greeted with a segmentation fault. At that moment, I just knew that things weren't going to be pretty. And I wasn't disappointed. It took me nearly an hour to figure out exactly what was going on and how to fix it. Apparently, the script had been modified when I had installed the original system so that it would run on the then-available version of CentOS. I vaguely remember doing that modification, and, had the system notes for that server not gone missing, I wouldn't have wasted so much time investigating the problem. I reversed the patches and was able to get the script to complete without error. I smiled to myself as the VMWare server software started, and I confirmed that both the Linux and Windows XP guests were running properly.

As I sat there patting myself on the back for dodging a bullet, my client attempted to do a backup of his QuickBooks data and encountered the error that had prompted my visit in the first place: there was insufficient disk space to perform the backup. At that moment, I realized that there were two disk-space issues. The first was on the server itself, which we had just remedied by adding two new drives. The second was on the virtual disk used by the XP guest. Apparently, QuickBooks does a bit of data collection on the local drive prior to writing the backup anywhere else, and the remaining space on the virtual drive was too small to hold these work files. No problem! I downloaded an ISO copy of the GParted live CD (think open-source Partition Magic) and went about configuring the QB Windows XP server to boot from it so that I could extend the virtual disk. This is where I entered the second circle of IT hell: "Configuration."

One of the charming aspects of VMWare is its propensity to change the names of its products on a regular basis, making it virtually impossible to hold discussions about them without all participants having access to an "Idiot's Guide to VMWare Product Names." Along with their name changes comes the equally charming propensity to change the method one would use to configure the host and guest instances. It took me only a short time to Google the correct method for VMWare Server 2.0.2, which happened to be by using a browser instead of special software. Excellent! Well, it was excellent until I actually tried connecting to the server at the appropriate port. I would get a connection and the correct tab title and then be treated to the Firefox whirlybird letting me know that the connection was open and data was being transferred. Eventually, the connection was reset by the server, and all I got was a blank page. @#$%! I spent an hour or so trying to figure out why the Tomcat server, which is how VMWare Server 2 gives browser access, was coughing up hairballs. I finally gave up. Knowing that VMWare has always provided a command-line suite to manage virtual machines, I used one of them to "power off" the XP guest machine. About the time I was browsing the help text to find the right command to assign the aforementioned ISO image to be the CD, it hit me... and thus began my introduction to the third circle of IT hell: "GUI operating systems."

In order to use the GParted CD to do the disk resizing, and later to boot to XP's GUI interface so as to load VNC, I would need to access the console of the machine. The way to do that? Via the Web-based configuration tool, which, of course, was broken. My solution to this conundrum was simple and based on one of the major advantages of virtualization: move the virtual machine elsewhere. In this case, I copied the virtual machine files to my laptop (only 12G worth of data), fired up VMWare workstation, booted the GParted ISO, and expanded the XP virtual drive. Once that step completed, I rebooted, this time directly into XP, waited for the requisite FDISK to complete, and was able to complete all of the tasks that prompted my visit in the first place. I shut down the XP guest, copied the files back to the original location, and then used the command line tools to start it. Success!

I was able to work around the problems I encountered during this process, but the whole thing isn't a case of "all's well that ends well." The fact that VMWare isn't functioning properly is still a problem that needs to be addressed. Sure, the two host virtual machines are working well, and with the additional of VNC I can readily access the Windows XP GUI, but I'm still unable to get to the console of either machine. This will come back to bite me (and this client) in the future. Had this machine been kept up to date, the problems would have been discovered earlier and they wouldn't have snowballed, thus turning a two-hour project into a ten-hour one. I've convinced this client that it's time to purchase a bigger server to replace this one. We'll re-purpose the older one. The new server will be a virtual host running either VMWare ESX or Linux/KVM, and I'll add a third guest to provide the services that the original server did. I've also convinced the client that next time we will need to budget some downtime for maintenance.

You know as well as I do that good IT practice calls for keeping your software (firmware included) up to date, so there's no need to continue to harp on that. I was lucky; I only had to wander through three circles of IT hell with this project. I'm convinced that there are nine, just as in Dante's Inferno, and I wrote this article as a reminder to myself that the time saved by not keeping things updated may end up being spent exploring the rest. If it ain't broke, don't fix it. Yeah, right.