The Exchange Intelligent Message Filter

Spam, or unsolicited commercial email (UCE), is an increasing annoyance for email administrators and users alike. Microsoft Exchange Server 2003 features increased antispam protection, offering a full array of filters that you can deploy to combat the bad guys. (For more details about Exchange 2003's antispam improvements, see the sidebar "Better Protection in Exchange 2003," http://www.winnetmag.com, InstantDoc ID 42683.) But filters rely on external input, such as realtime blackhole lists (RBLs), and you need a reasonable degree of expertise to configure Exchange 2003's filters correctly. Even then, keeping pace with spam is almost impossible because spammers frequently change their tactics to evade detection. For example, spammers switch domains to avoid RBLs and constantly play with the text of their messages or use foreign languages to avoid recognition by antispam tools.

To help you better protect your Exchange systems, Microsoft Research has developed SmartScreen Technology, a patented machine-based learning technology that can recognize the distinguishing characteristics of both legitimate email and spam, based on a huge collection of messages that Microsoft gathered from inside the company and from customers. Early versions of SmartScreen Technology appear in MSN 8, Microsoft Hotmail, and Microsoft Office Outlook 2003, but the technology's first major appearance is in the Microsoft Exchange Intelligent Message Filter (IMF), an Exchange 2003 add-on that Microsoft released in May 2004. IMF's implementation is designed to detect spam that tries to enter an organization through its Exchange SMTP connectors. Microsoft supports IMF only on standalone servers running Exchange 2003 Enterprise Edition or Exchange 2003 Standard Edition; you can't install IMF on a clustered Exchange 2003 server or on earlier Exchange versions. (You can deploy IMF on Exchange 2003 gateway servers to protect Exchange 2000 Server or Exchange Server 5.5 mailbox servers, but this type of implementation delivers only half the potential benefit of IMF.) Microsoft originally planned to make IMF available only to customers who had licensed Exchange 2003 under Software Assurance (SA), but later reconsidered that decision and, at Microsoft TechEd 2004, released IMF to all Exchange 2003 customers. Is IMF a worthwhile addition to your Exchange 2003 deployment—or perhaps even worth an upgrade to Exchange 2003? You'll be better equipped to make that decision if you know a bit about how IMF works and how you might benefit from deploying it.

The ABCs of IMF
IMF scans messages and assesses whether they're spam or legitimate email, according to certain characteristics that Microsoft defined after studying a huge sample of real-life messages. IMF depends on intelligence (i.e., code) and data (e.g., spam patterns and indicators) to decide whether a message is spam. (Other antispam products use similar Bayesian filtering and heuristic-based detection techniques.) Given that spamming techniques change constantly and that spammers will attempt to discover how IMF works and how best to avoid being detected by it, Microsoft will need to update IMF intelligence and data regularly. Microsoft plans to automate this process at some point, but initially, you'll need to download IMF updates from the Microsoft Web site and manually apply the updates to maintain IMF’s effectiveness.

IMF works only with SMTP connectors. This limitation is reasonable because X.400 and other messaging connectors link messaging systems that are well known to one another and so aren't likely to be spam sources or targets. SMTP, however, is the common tongue of the Internet; spammers operate by sending messages to SMTP addresses that they harvest from public repositories (such as newsgroups or Web sites) or that they generate for targeted domains by using the firstname.lastname@company.com formula.

IMF scans messages after they pass through any other filters that you've deployed on the Exchange servers that host SMTP connectors to the Internet. (Exchange applies connection, recipient, and sender filters, in that order, before passing messages to IMF.) IMF evaluates messages against its characteristics set and supplies the outcome as a rating called a spam confidence level (SCL), which ranges in value from 1 to 9. (Exchange uses two other values for its own mail routing: 0 indicates legitimate email and -1 indicates that the message originated inside the Exchange server's organization and requires no further validation. You can find information about programmatically manipulating SCL values in the Microsoft Exchange Server 2003 Software Development Kit's—SDK's—Solutions section, at http://msdn.microsoft.com/library/en-us/e2k3/e2k3/_e2k3_solutions.asp.) The lower the SCL, the less likely a message is to be spam. Generally, a message that receives an SCL higher than 5 is guaranteed to be spam. Exchange stores the SCL as a message property. (Out of the box, the Exchange 2003 Store supports this property and adds new attributes to the AD schema to control UCE processing but doesn't provide a UI that you can use to set SCL thresholds or to control what happens to a message in response to its SCL rating. Even when you install IMF, the degree of control you have over a message’s destination is limited: IMF simply let's you tell it whether to drop or accept a message according to the message's SCL rating. However, third-party utilities or future versions of IMF could work with the SCL property to provide a wider range of processing options.) If a message's SCL meets a predefined threshold, IMF suppresses the message at the gateway; otherwise, the message passes through the gateway, and the mailbox stores perform further processing based on the message's SCL rating.

Deployment Decisions
You can deploy Exchange servers that host SMTP connectors as standalone servers in a demilitarized zone (DMZ), or you can deploy them behind a firewall, in which case you need to install other servers to operate in the DMZ and to provide an initial SMTP connection to the Internet. (Many companies deploy Linux or UNIX systems, running email Message Transfer Agents—MTAs—such as sendmail or Postfix, in the DMZ to provide basic handling of messages entering and leaving the organization.) Deploying Exchange in the DMZ is unusual, but the increased level of security in Windows Server 2003 and Exchange 2003 makes doing so a reasonable option.

If you deploy Exchange in the DMZ, you should put the servers in a separate Exchange organization to reduce the potential for a security breach. Ideally, these servers should be under the control of separate administrators and shouldn't host mailboxes. The servers' sole function should be to accept incoming messages; to check for viruses, spam, or unwanted senders; and to pass (or relay) the messages to servers behind the firewall for delivery to mailboxes. Because you can have only one Exchange organization per Active Directory (AD) forest, this setup necessitates separate forests—and added cost. When you deploy IMF in such a setup, you must enable cross-forest authentication so that Exchange can send SCL rating information between organizations. The connectors that link the forests must use authenticated accounts (authenticated connections let Exchange transmit extended message properties, including the SCL rating, between forests) instead of making unauthenticated connections, which is the norm. If you decide to rely only on IMF gateway filtering rather than using Store filtering, you can revert to unauthenticated connections.

The more likely scenario is to run IMF on one or more Exchange gateway servers behind the firewall and to use SMTP connectors to link these servers to the DMZ server. If you operate other email systems alongside Exchange in your messaging infrastructure and use SMTP to pass messages among the various systems, you should also deploy IMF on any Exchange servers that host the SMTP gateways to the other mail systems to ensure that IMF can catch any spam that arrives through those systems.

Know Your Limits
IMF installation is simple, and the only signs you'll have of the new add-on are the addition of an Intelligent Message Filtering tab in the Exchange System Manager (ESM) Message Delivery Properties dialog box (which you can access through the Global SettingsMessage Delivery node) and a new ESM Intelligent Message Filtering node (under the ProtocolsSMTP node for the Exchange server that's running IMF). After IMF installation, open ESM and open the Message Delivery Properties dialog box, then go to the Intelligent Message Filtering tab, which Figure 1 shows, to set SCL thresholds for gateway and Store processing and to set an action to occur when spam is blocked at the gateway. (Note that the Store threshold, which you set under the Store Junk E-mail Configuration section, must be lower than the gateway threshold, which you set under the Gateway Blocking Configuration section.) Like other Global Settings, these settings apply to all servers within the Exchange organization.

The gateway SCL threshold (sometimes referred to as the block threshold) controls the action that Exchange takes on the server that hosts the SMTP connector. You'll need to perform a balancing act when determining this setting. If you set the threshold too low—say, 2—you run the risk that legitimate messages will be treated as spam. If you set the threshold too high—say, 7—your users might still get more spam than they care for. Every company will have a different tolerance level, and some believe that it's better to accept some spam to be sure that every legitimate message gets through as quickly as possible. Others believe that it's better to filter as much spam as possible and to have a reliable process for dealing with false positives. The best idea is to start with thresholds that are slightly higher than you think you need, keep an eye on the volume of false positives that occur during the first couple of weeks, then reduce the threshold until you arrive at a steady state at which IMF blocks the maximum amount of spam possible without generating an excess of false positives.

Your options for determining what the gateway server does with a message that has an SCL rating equal to or greater than the gateway threshold are Archive, Delete, No Action, and Reject. When you select Delete, IMF immediately drops the message. The No Action setting causes Exchange to let the message pass the gateway, although the message might still be rejected during store-level processing. Reject causes the gateway to return the message to its originator. You can use the Archive setting to archive messages so that you can check for false positives. (See the sidebar "Archiving Dos and Don'ts," http://www.winnetmag.com, InstantDoc ID 42685 for information about dealing with this setting and with archived messages.)

The Exchange gateway server takes no action for messages that have an SCL lower than the threshold; Exchange routes those messages to the mailbox server that hosts the destination mailbox. When the messages arrive at the mailbox server, the transport engine delivers them to the Store, which (assuming that the mailbox server is an Exchange 2003 server) checks the messages against the Store threshold and against the Blocked Senders and Safe Senders lists that users have configured through Outlook 2003 or Outlook Web Access (OWA) 2003. If the SCL rating is equal to or lower than the Store threshold and the message sender isn't on the recipient's Blocked Senders list (or the Store can’t access that list—for example, because the recipient is using an earlier version of Outlook or OWA), the Store delivers the message to the user’s Inbox. If the sender is on the Blocked Senders list, the Store takes the action defined in the user’s Junk E-mail options (e.g., deliver the message to the user's junk mail folder, delete the message permanently). If the SCL rating is higher than the Store threshold, the Store checks the user’s Safe Senders List. If the Store finds the sender’s email address on that list (or can't find a Safe Senders list for the mailbox), it delivers the message to the user's mailbox. Otherwise, the Store handles the message according to the user’s Junk E-mail options.

Apply Yourself
After you've configured the global IMF settings, you have to activate IMF on the SMTP virtual servers that handle message traffic. In concept, this operation is the same one you use to enable connection, sender, or recipient filtering—you just perform this task through a slightly different interface. The primary difference is that you can apply IMF to multiple SMTP virtual servers simultaneously, whereas you apply other filters to virtual servers one by one.

As I mentioned earlier, after you install IMF on an Exchange server, a new Intelligent Message Filtering node appears in ESM under that server's ProtocolsSMTP node. The Intelligent Message Filtering node's Properties dialog box lists all the SMTP virtual servers that are active on the Exchange server and indicates whether IMF is active on each virtual server. (Figure 2 shows an example of what you'll see on the vast majority of Exchange 2003 servers: One default SMTP virtual server associated with TCP port 25 and handling all IP addresses. Each physical Exchange server can support multiple SMTP virtual servers—assigned to different ports and handling different ranges of IP addresses—but making such changes to the default configuration is usually unnecessary.)

Get More Information
Behind the scenes, IMF writes a number of new events to the Windows Application log when specific actions occur. For example, event ID 7512 provides information about messages that the gateway has rejected or deleted. More interestingly, IMF writes event 7515 whenever it can't process a message (because, for example, the message was malformed or corrupted in some way). Event data tends to be sparse, however, so to gain insight into IMF’s activity level, you can use Performance Monitor to view the performance counters under the MSExchange Intelligent Message Filter object. These counters give you a total count of scanned messages, scan rate per second, and totals for rejected, deleted, and archived messages. A separate counter reports the percentage of total scanned messages that IMF has marked as spam; another counter displays the percentage of spam received in the past 30 minutes.

The best approach to stopping spam is to implement multiple layers of defense and to have as much knowledge as possible about what your organization is up against. IMF can be a useful addition on both fronts. However, you might want additional protection, in which case third-party alternatives might be in order. For more information about your other options, see the sidebar "IMF Alternatives," http://www.winnetmag.com, InstantDoc ID 42684.)