Updating SMTP Support

In step with the move to support Internet protocols in all its products, Microsoft has included Internet Information Services (IIS) 5.0 with Windows 2000 Server, thus providing built-in SMTP capabilities to various applications, including Microsoft Exchange 2000 Server. Exchange 2000 extends Win2K's basic SMTP capabilities and introduces a new SMTP-based routing engine, which replaces Exchange Server 5.5's X.400-based Message Transfer Agent (MTA) and optional Internet Mail Service (IMS). This transition marks a fundamental increase in Exchange's SMTP support.

If you're considering migrating to Exchange 2000 or have already begun to use it in your organization, you'll benefit from learning how Win2K and Exchange 2000 implement SMTP. I visited this topic in "Exchange 2000 and SMTP," March 2000, but now that both Win2K and Exchange 2000 are on the shelves, a more extensive discussion is in order. Exchange 2000's support of SMTP extensions, increased emphasis on Routing Group Connectors (RGCs) and SMTP connectors, and use of the new routing engine and the Transport Core all highlight the application's—and Microsoft's—dedication to SMTP.

Win2K: Integrating Support

Enabling SMTP capabilities on a Win2K server doesn't take much work on your part. During Win2K installation (or during an upgrade from Internet Information Server—IIS—4.0 to IIS 5.0), simply configure IIS 5.0 to include SMTP and Network News Transfer Protocol (NNTP). A basic SMTP service then becomes operational on the Win2K server. Exchange 2000 uses Win2K's IIS Internet protocol stacks to implement protocol virtual servers. A protocol virtual server combines an IIS protocol stack, an IP port number, and an IP address. The default SMTP virtual server combines the SMTP protocol stack, port 25, and the host server's IP address.

A Win2K server can support multiple protocol virtual servers so long as you allocate a unique IP port number for each virtual server (Microsoft recommends, however, that you support no more than two virtual servers per protocol on a physical server). You'll rarely need to create multiple SMTP virtual servers within your organization, though; rather, you can create multiple SMTP connectors, each with a separate set of parameters, all running under the control of one SMTP virtual server. (For example, you can create one SMTP connector to process large messages according to a predetermined schedule and another connector to immediately send messages that are smaller than a specified size.) All the SMTP connectors that run under the control of a protocol virtual server share a common set of properties, such as the Badmail directory (i.e., the place where the virtual SMTP server puts any message that it deems to be undeliverable). The Microsoft article "XADM: How to Configure an SMTP Connector" (http://support.microsoft.com/support/kb/articles/q266/3/17.asp) explains the procedure for configuring SMTP connectors.

The basis for Win2K's SMTP service is Collaboration Data Objects for Windows NT Server (CDONTS), a set of code that Microsoft's Exchange Server engineering group developed. When you install Exchange 2000, its installation procedure upgrades the Win2K SMTP service through a new set of transport and protocol events (which add features such as support for mailboxes and distribution groups) and through such advanced functionality as Link State Routing. Unlike Exchange 5.5, which requires you to configure the IMS separately after Exchange installation, you need to configure the default SMTP virtual server only if you want to change its default properties.

When you install Exchange 2000, the management of SMTP and NNTP virtual servers transfers from the Microsoft Management Console (MMC) Internet Services Manager snap-in to the MMC Exchange System Manager snap-in, in which the servers appear under the appropriate protocol. (For instructions about when and how to return control of the virtual servers to the Internet Services Manager snap-in, read the Microsoft article "XADM: Exchange 2000 Setup Removes SMTP and NNTP from Internet Services Manager" at http://support.microsoft.com/support/kb/articles/q274/3/45.asp.) Before you use the Exchange System Manager snap-in to make changes to the protocol virtual servers, make sure you understand how the IIS metabase (which holds information about the protocol virtual servers) synchronizes with Active Directory (AD—which holds routing and other configuration information). Otherwise, you run the risk of overwriting previous changes to the servers. To change an IIS component that Exchange uses, you should use the Exchange System Manager snap-in, then let AD's synchronization process update the IIS metabase. For an overview of the synchronization process, see the Microsoft article "XGEN: General Information on Directory Service/Metabase Synchronization in Exchange 2000 Server" (http://support.microsoft.com/support/kb/articles/q240/1/05.asp).

Exchange 2000: Extending SMTP

SMTP's basic approach to mail exchange was satisfactory in the Internet's early days, when people sent only short and rudimentary text messages. Today, though, people often need to send huge messages to long distribution lists (DLs)—and fast. Exchange 5.5 supports several SMTP service extensions, which Microsoft refers to as Extended SMTP (ESMTP), that add certain features to additional SMTP verbs (e.g., ETRN) that help the protocol process bigger messages faster.

Exchange 2000's support of additional ESMTP extensions—specifically Chunking, 8bit-MIMEtransport, and Pipelining—represents a crucial advance over Exchange 5.5. Internet Engineering Task Force (IETF) Requests for Comments (RFCs) provide definitions of these extensions and the ESMTP verbs that they introduce. (You can find the text of the IETF RFCs at various Web sites, including http://www.rfc-editor.org.) Microsoft has also created two ESMTP verbs—X-EXCH50 and X-LINK2STATE—that are specific to Exchange 2000. To determine which extensions an Exchange server supports, you can use the SMTP verb EHLO, as the sidebar "EHLO: Exchange 2000 Calling," page 94, explains.

Chunking. The traditional SMTP transfer consists of a series of DATA verbs in a format broadly equivalent to a series of text lines. Each line terminates with

CR LF . CR LF

This method works well enough when you have a limited number of lines to transmit but incurs a huge overhead as attachments grow. Chunking, which introduces the BDAT verb, lets Exchange 2000 transfer message content between servers in a continuous stream from beginning to end—a much more efficient method.

8bit-MIMEtransport. In today's messaging environment, in which users commonly attach multimegabyte Microsoft PowerPoint presentations, video, or audio clips to their messages, binary data transfer is essential. Most applications that transmit graphics by means of email use 8-bit MIME standards to encode video and audio content. The 8bit-MIMEtransport extension enables support for this content and avoids the overhead that traditional SMTP servers expend translating message content back and forth between 8-bit and 7-bit MIME.

Pipelining. The Pipelining extension also increases message throughput. The original SMTP protocol calls for the receiving server to acknowledge each SMTP verb that it receives. The sending server can't proceed to the next verb until it receives that acknowledgment. (This implementation was necessary back when computer networks' connectivity was less reliable than it is today.) The new Pipelining extension lets SMTP servers send out a stream of verbs without waiting for acknowledgments, thereby speeding up message transfer.

X-EXCH50. Exchange 2000 uses the X-EXCH50 extension for SMTP transfers of system information (e.g., public folder replication updates) to another server within a routing group or across an SMTP connector. You can impose message-size restrictions for a connector, but you don't want to prevent system messages that exceed your set limits from getting through. To solve the problem, Exchange 2000 issues the X-EXCH50 command before it attempts to transfer system messages. This command tells the connector to let the messages pass. See the Microsoft article "XGEN: Dealing with System Messages Between Exchange 2000 Server Computers" (http://support.microsoft.com/support/kb/articles/q233/2/06.asp) for additional information about this ESMTP extension. (Note that this article incorrectly refers to the X-EXCH50 extension as X-EXCH60.)

X-LINK2STATE. Communications in a messaging network vary constantly: Connectors and servers go on- and offline; new message paths become active. Exchange 2000 uses Link State Routing to exchange knowledge of routing conditions and to maintain a dynamic picture of these conditions for the entire messaging network. (For a detailed discussion of Link State Routing, see the sidebar "Link State Routing.") Exchange 2000 servers use the X-LINK2STATE command to send link-state updates among routing groups across RGCs and SMTP connectors. (You can transfer link-state information across an X.400 connector when you use a field in the header of a system message.) However, as companies complete their migrations and move into Exchange 2000 native mode, RGCs will likely replace X.400 connectors; thus, SMTP will become the only transport protocol used within these organizations. For a discussion of all three connectors, see the sidebar "Making the Connection," page 98. The sending server starts with an EHLO query to ensure that the target server understands the X-LINK2STATE verb. The sending server then transfers the update information in a highly compressed format. (Because Exchange 2000 considers these SMTP transfers to be the same as any SMTP transaction, it uses port 25 to transfer data between routing groups. Exchange 2000 uses port 691 to send link-state updates within a routing group.)

Routing and the Transport Core

So, Exchange 2000 extends the basic SMTP capabilities that Win2K provides. But how has the actual routing of messages changed in Exchange 2000? Together with the routing engine, three other components—the Advanced Queuing engine, the Message Categorizer, and the Store Driver—process the message. These components form the Exchange 2000 Transport Core.

The Routing Engine service operates as a process within IIS and involves several other services in message routing. For Messaging API (MAPI) and Outlook Web Access (OWA) clients, the Information Store service deals with messages to and from local mailboxes. The IMAP4 and POP3 services send messages through an SMTP connection and use their own protocol to retrieve messages from the Exchange Store. The MTA service provides X.400 services and connectivity to messaging systems such as Microsoft Mail (MS Mail), Lotus cc:Mail, and Lotus Notes. The SMTP service accepts incoming SMTP messages and sends outgoing messages, including those submitted by POP3 and IMAP4 clients.

The Information Store, MTA, and SMTP services submit messages to the Transport Core. Exchange 2000 uses the Exchange Installable File System (ExIFS) to write incoming SMTP messages first to disk, then to the Store. After submission to the Store, the routing engine places messages on a queue and the Advanced Queuing engine performs several checks, including address resolution (the engine detects irresolvable addresses) and determination of recipient or mailbox restrictions. (Address validation requires a lot of interaction with a Global Catalog—GC—server, which contains information about all the mailboxes and groups inside the organization. This requirement is one reason that Exchange 2000 deployments need more GC servers in place than do Win2K implementations that support other applications.) The Advanced Queuing engine also processes the expansion of distribution groups. The engine then generates nondelivery notification messages for rejected messages and places all other messages into the categorization queue.

The routing engine can now fire any registered transport events (i.e., specific actions that occur during message transport, such as checking messages for viruses or content restrictions). After the routing engine processes all transport events, the Categorizer decides the message's next step. The Categorizer can place the message on a local queue for delivery to a local mailbox or the MTA, place it on a queue for transmission to another Exchange 2000 server or routing group, or deliver it through an SMTP connector to an external domain.

The Store Driver handles messages that the Categorizer queues for delivery to local mailboxes or the MTA (which uses special hidden folders to process incoming and outgoing messages). The Store Driver moves messages into Store folders, including user Inboxes and hidden folders. The routing engine then fires any Store-Driver events (i.e., events associated with items already committed to the Store database).

Servers that host an SMTP connector display a special SMTP mailbox in the default mailbox store, as Figure 1 shows. Don't assume that this mailbox is temporary or unnecessary and delete it—you'll halt all SMTP connections! The SMTP mailbox is a placeholder to which the Store Driver submits messages that come in through the connector en route to a local mailbox or to the MTA for transmission through an MTA-hosted connector to another server.

The routing engine manages, on per-domain destination queues, messages that the Categorizer queues for SMTP delivery to other Exchange 2000 servers or to SMTP domains. The routing engine maintains a separate queue for each destination (e.g., Microsoft.com, another Exchange 2000 server). You can view these queues, as well as the queues that the Transport Core components use, through the Exchange System Manager console. Figure 2, page 97, for example, shows that five messages are pending delivery to QEMEA-ES0. Exchange 2000 usually processes messages quickly, so you should investigate this type of message backlog because it indicates that a connector, server, or network link isn't working properly.

After the Categorizer processes messages, the Routing Engine moves them to link queues, which essentially are roadways off the server en route to the message's final destination. The routing engine maintains link queues on a per-hop basis. For example, if messages to a Microsoft.com addressee need to move to another Exchange 2000 server in a different routing group (because that routing group hosts an SMTP connector that can service Microsoft.com), the messages will move from the Microsoft.com domain queue to a link queue for the routing group in which the SMTP connector resides. Messages might flow from multiple domain queues to one link queue—a very different implementation from the IMS, which holds all messages on one queue throughout processing.

To learn more about the way messages travel through your Exchange organization, you can use the Winroute utility. For information about this handy tool, see the sidebar "Go with the Flow."

What About the MTA?

The MTA is still present on Exchange 2000 servers. However, the new routing engine has taken over a lot of the MTA's previous duties, such as routing, expansion of DLs, and fan-out procedures (i.e., the creation of an optimum number of message copies to go to various servers).

The exception to this statement is fan-out for queues that the MTA hosts, such as queues for Exchange 5.5 sites, X.400 connectors, and any other connector based on the Microsoft Exchange 5.5 Development Kit (EDK). Examples of these connectors include MS Mail, Lotus cc:Mail, and Lotus Notes, as well as most modern fax connectors. The routing engine determines that the MTA is the next hop for messages to these destinations and hands the message to the MTA. The MTA examines the message header and generates copies to the necessary connectors or uses a remote procedure call (RPC) to transmit the message to another MTA. During this process, the MTA also performs any required conversions from X.400 P2 or P22 format to Microsoft message database encoding format (MDBEF). For example, the content of an incoming message from a foreign X.400 system will be in P2 or P22 format, depending on the set of X.400 recommendations that the foreign mail system supports. When the MTA receives the message, it detects the P2 or P22 body part and converts it to MDBEF.

New and Improved

In most cases, Exchange's switch from X.400 to SMTP is successful and messages flow freely. Exchange 2000 Service Pack 1 (SP1) deals with several concerns that came to light after Exchange 2000 shipped, and Microsoft is sure to provide more fixes in SP2 (which should ship toward the end of the year). Whether you're considering Exchange 2000 or have already deployed it, devote some time to investigating the Microsoft Knowledge Base and other resources for information about the product's new SMTP capabilities and other changes. Such research can pay big dividends.