Last February, IBM took a look at what the Internet had wrought and discovered that 76% of all email sent through the Internet could be considered spam. In addition, one in every 46 emails (about 2%) contained a virus, a Trojan horse, or some other form of malware.

The Good News

Think those numbers are outrageous? Well, the good news is that the spam numbers were actually down from January, when IBM recorded that 83% of all email sent was spam. Imagine that: Only 17 % of email sent was actually valuable. No matter how you cut those numbers, the plain fact is that the Internet email network--the way most of us do business today--is rife with junk. And, as users, we have to deal with what we get, day in and day out.

IBM estimates that spam is currently costing U.S. companies up to $17 billion a year in lost productivity. Why? Because somebody has to make decisions about all the spam email. And the biggest problem of all is that recipients cannot automatically validate a sender's address and reject the unsolicited communications. We have to read each one.

Why can't we stop this absurd situation? After all, didn't congress pass an anti-spam law? Well, yes, Congress did. But legal penalties clearly are not sufficient to halt the continual technological hijacking of Internet email. It's just too difficult to track down the culprits.

Why Spam Is So Difficult to Control

The technical cause of this problem is the relatively unsophisticated Internet protocol called Simple Mail Transfer Protocol (SMTP), a protocol that grew out of the early days of Internet email implementation. SMTP is the "sending" protocol of Internet email servers, and though it does an exceptional job, it is not very picky about how it identifies senders.

For instance, SMTP does not require that email indicate who the actual sender is. I can send email with a fake "From" address. This is called "spoofing," and it's the main means by which spammers are able to produce so much garbage through our email systems. However, the IP address of the sender is indeed captured by the SMTP mechanism, and for a while, this seemed like a good enough means of identification, since every IP address is unique.

However, hackers have turned obfuscating their identities into an art. Using a combination of Trojan horses and email viruses to infect unsuspecting Internet users with small, customized SMTP server programs, these hackers have turned thousands of PCs connected to the Internet into zombie spam distribution centers. These machines now represent the infrastructure of an underground spamming network that is largely invisible to authorities. The zombie machines are controlled surreptitiously across the Internet by the owners of the spam networks, and the owners of the zombie machines often don't even know their machines are a part of this underground distribution network. This is why current legal penalties are not working: The technology for hiding the real sender's identity is stronger than SMTP's ability to certify that identity.

How much easier it would be if SMTP contained a requirement for security certificates, much as Lotus Notes email does for internal, non-SMTP mail. By using such certificates, it would be relatively easy for administrators to track down people who are abusing the system and reject their messages from users' inboxes.

Unfortunately, without some sort of built-in SMTP identity check, companies today are forced to use spam filtering mechanisms to sort the garbage from real messages. And too often, those filters fail. That is, until recently.

IBM Introduces FairUCE, a New Anti-Spam Technology

It is against this background of continual email abuse that IBM has introduced a new anti-spam technology to help developers and Internet email system administrators reduce the cost and security risks associated with spam. It's a technology designed to make existing spam filtering solutions more effective. IBM is calling this technology "Fair use of Unsolicited Commercial Email," or FairUCE.

FairUCE is a spam filter technology that stops spam with a novel approach: It figures out a way to check the sender's identity instead of filtering the actual content of each email communication. According to IBM, FairUCE stops the vast majority of spam without the use of a content filter and without requiring a "probable spam" or "bulk folder" that needs to be checked periodically.

IBM says it is one of the first spam filters to use this technique of validating the sender's identity--rather than email content--to determine a message's legitimacy.

Better Than Content Filtering

So why is identity validation better than content filtering?

Content filtering examines what's in the email, checking for key words, phrases, or patterns of content. It's highly sophisticated but prone to mistakes and in need of continual maintenance.

For instance, AOL estimates that spammers often respond within four hours to a change in its content filter. If a message is getting blocked by a content filter, a spammer can usually figure out a new means of slipping the message past the monitor. Moreover, content filters require a great deal of processing horsepower, using complex techniques such as Bayesian filtering, heuristics, and digital fingerprinting. Using a content filter to block spam is equivalent to hiring a policeman to monitor a room to remove people who use specific offensive language: People who want to get their offensive message across quickly learn to use other words to communicate their intent.

That's why FairUCE seems like a better method. According to IBM, FairUCE doesn't try to filter the content of a message. Instead, it attempts to track down who actually sent the message, gathering information about the message transmission, and then selectively checks the validity of that sender.

How Does FairUCE Work?

According to IBM, FairUCE tries to find a relationship between the sender's domain and the IP address of the client delivering the email, using a series of cached DNS lookups. IBM says that for the vast majority of legitimate email, this is easily accomplished. If the appropriate relationship can't be verified, FairUCE then attempts to find one by sending a user-customizable challenge/response. According to IBM, this alone catches 80% of spam and very rarely challenges legitimate email.

Finally, if a valid relationship can be found, FairUCE screens the message against the recipient's whitelist and blacklist--and checks the domain's historical reputation within administrative blacklists--to determine whether to accept, reject, challenge on reputation, or present the user with a set of whitelist/blacklist options.

IBM says future versions of FairUCE will also incorporate Sender Policy Framework (SPF) or similar sender identification systems, as well as use a real domain reputation system.

Implementation of FairUCE

The FairUCE concept is currently implemented as an SMTP proxy that runs between multiple instances of Postfix on Linux. IBM says QMail and Sendmail support are being considered. However, IBM says it should be possible to use existing mail servers on the inside of the SMTP proxy.

What's interesting about IBM's announcement is that end users can't install FairUCE. It's not a "product" but a technology for use by email administrators and by developers of spam filters. (Find more information about FairUCE at IBM's alphaWorks.)

Comparing the Microsoft Partnering Strategy

Now compare IBM's approach of creating a new anti-spam technology with Microsoft's recent announcement that it would buy Sybari Software.

Sybari makes anti-virus and anti-spam software that integrates with Microsoft Exchange, Lotus Domino, and various SMTP gateways across multiple operating system platforms. Sybari was a good partner for Microsoft (as well as other companies), bringing a quality solution to a deeply technical problem. Sybari was gaining significant momentum as a company, and analysts were predicting that soon it would have a stock IPO. Microsoft's acquisition of Sybari has quashed that prospect.

Nonetheless, one might hope that Microsoft's acquisition would continue to foster some kind of technological partnership across all operating system platforms. After all, the problems of Internet spam and viruses are global in nature, requiring coordinated efforts in a standards environment. For instance, Sybari currently makes Linux and UNIX versions of its products.

Unfortunately, however, Microsoft has already announced that it will soon discontinue sales of Sybari products on non-Microsoft operating system platforms. In addition, analysts are predicting that Microsoft will phase out support for non-Microsoft email systems by 2008.

In other words, the Microsoft strategy for partnering seems to be "find the best company with the best solution, acquire it to control its technology, and then make its products proprietary to work only with Microsoft products."

Now, as a software company with only one operating system platform, Microsoft's strategy probably makes good business sense. However, as a partnering strategy, it sends a lot of mixed messages to both customers and business partners. It's a strategy of exclusivity. And while being exclusive may seem like a great opportunity for some companies, it's a problem when customers are seeking technologies to solve global, cross-platform problems.

IBM's Partnering Strategy

IBM's partnering strategy appears to be in stark contrast to Microsoft's. IBM clearly tells its Business Partners that it is not in the "solution space." Instead, IBM provides technological foundations for Partners to use in developing product solutions. IBM also provides marketing support to its Business Partners, helping them crack open new markets and develop lasting customer relationships to build business depth.

In other words, instead of buying up partners to monopolize technology, IBM seems to understand that growing its Business Partner relationships is the key to a win-win scenario for everyone.

Best Practices in a Global Internet Community

There's no question that Microsoft fosters good relationships with many of its business partners. What's not always so clear is how well Microsoft's business strategies work for the betterment of its customers.

As we continue to struggle with the intricacies of Internet expansion--embracing new technologies while fighting off evolving security threats--it gives us pause to reflect that in a truly connected world it should no longer matter to what operating system we are sending our email, nor what operating system is sending mail to us. Who cares if it's IBM's i5/OS SMTP Gateway, Lotus Domino, Microsoft Exchange, or QMail?

Reducing spam is a global requirement, just as SMTP is a global protocol. If email is to continue to provide significant value to our business communications, proprietary software and exclusive vendor relationships simply become obstacles to doing better business.

Thus, it seems that IBM offers best practices for fostering good partner relationships in this new global environment by providing open technology like FairUCE to developers and Business Partners on all operating system platforms. If Microsoft's acquisition of Sybari Systems is any measure, Microsoft has a lot to learn about partnering from IBM.

Thomas M. Stockwell is Editor in Chief of MC Press Online, LP.