Securing SMTP Email Traffic

Because the SMTP standard sends email without using encryption or authentication, every message you send is exposed to view. Client-side solutions such as Secure MIME (S/MIME) or pretty good privacy (PGP) can solve this problem, but they require your users' involvement. A better place to focus your security efforts is on securing SMTP traffic. If you can secure SMTP, you'll go a long way toward providing 100 percent security for mail traffic that originates or terminates at one of your servers.

Microsoft Exchange Server offers several tools for securing email traffic. One way to secure SMTP is to require the use of Secure Sockets Layer (SSL) for SMTP connections. However, that approach raises a problem. By default, all SMTP servers use port 25. But if you use SSL on port 25, non-SSL servers won't be able to connect through that port. And if you use a nonstandard port number, other servers won't be able to find your servers.

You can work around this problem. The STARTTLS verb (part of the Extended SMTP—ESMTP—command set) lets an SMTP client and server negotiate the use of Transport Layer Security (TLS) for an SMTP connection. Each end of the connection can choose to authenticate the other, or the TLS connection can be used purely for privacy. Either way, this approach offers three important advantages.

You must remember one important caveat, however: TLS doesn't protect messages from end to end. In other words, it doesn't protect messages that are in storage or traveling from client to server (unless the client also supports TLS). TLS protects the message only as it passes between two servers that both support TLS.

Requesting an SSL Certificate
Before you can begin to use TLS with SMTP, you must obtain and install a certificate for your SMTP server. If you've already set up your own certificate authority (CA), you'll find that requesting an SSL certificate is quite simple; if not, you'll need to save the certificate request to a file and deliver it to your preferred CA. (For a detailed explanation of how to set up and use the Microsoft CA or how to use an external CA for your public key infrastructure—PKI—see the Certificate Services topic in Windows Server Help.) I assume that you have access to a CA of some sort and want to request and install an SSL certificate for use with Microsoft Outlook Web Access (OWA), IMAP, POP, or SMTP.

The basic mechanics of certificate issuance are the same for all these protocols, although I deal with only SMTP here. You initiate the certificate request process from Exchange System Manager (ESM). In ESM's treeview pane, navigate to your server, then to Protocols. Select SMTP, then right-click the virtual server and select Properties. On the Access tab, you'll see the Certificate button, which is enabled whenever you run ESM on an Exchange server. Because the private key associated with the certificate is generated on the local machine, generating a certificate from your administrative workstation doesn't make sense.

You can use Internet Services Manager (ISM) 5.0 to request a certificate for use with OWA. However, using ESM to request POP and IMAP certificates is easier because you can request them from the virtual server Properties dialog box, so that's what we do here. To request a certificate, simply click Certificate.

Getting a Certificate
When you click Certificate on a server that doesn't have a certificate for the specified protocol, ESM launches the IIS Certificate Wizard. The wizard will look familiar if you've ever used Microsoft IIS to request an SSL certificate. To request a certificate for Exchange to use, you must have administrative privileges on the Exchange server and the account you use must have permission to request the certificate from the CA.

After clearing the welcome screen, you need to specify what you want to do. If you're requesting a certificate for a virtual server that doesn't currently have one, your choices are to attach an existing certificate to the virtual server, import a backed-up certificate, or request a new certificate. If you're modifying an existing certificate, the wizard asks you to choose between renewing the certificate, removing it from the virtual server, and requesting a new one. Let's assume that you want to request a new certificate. After you click Create a new certificate, the process works as follows:

Up to this point, the procedure is the same whether you request a certificate from an online or offline CA. After you provide your geographic information, an online request prompts you to select the CA you want to use from a list of CAs visible to the requestor. (If the CA you want to use isn't listed, you can't submit a certificate to it.) Select a CA, then click Next. You'll see a summary screen of the parameters you've requested for your certificate; clicking Next again sends the request. When the request is successful, the certificate is installed automatically and you can begin configuring TLS.

Using a Third-Party CA
If you selected Prepare the request now, but send it later to submit a certificate to a CA outside your network or to a standalone CA that isn't integrated with Active Directory (AD), your submission process will be a bit different. After you provide geographic information, the wizard lets you choose where to save the request file. The request file is a plaintext, base-64­encoded PKCS #10 request. After saving the file, you need to submit it to your CA, typically through a Web-based interface. The details of the interface depend on the CA; for example, the interfaces for Thawte, VeriSign, and Comodo's InstantSSL look and behave somewhat differently from one another. Most vendors' Web sites prominently feature a set of instructions for submitting a request.

Microsoft's CA can also accept PKCS #10 requests through its Web interface. Let's walk through the process of using this CA to request and install a certificate.

You're not finished yet. As the Submission Confirmation page tells you, you need to return to the CA to check the status of the request and retrieve the issued certificate. (Depending on whether you're using an enterprise or standalone CA, the CA might automatically issue the certificate or require administrator intervention. For information about the difference between enterprise and standalone CAs, see the Certificate Services documentation in Windows Server Help.) To do so, return to the CA Welcome page at http://servername/certsrv, where servername is the name of your server. Select Check on a pending certificate, then click Next. The CA lists all known pending requests. Select yours, then click Next.

What you see then depends on whether the CA has issued your certificate. If it hasn't, the CA Web page tells you that the request is still pending. In that case, you'll need to approve the request (or have it approved). For an explanation of how to use the Microsoft Management Console (MMC) Certificate Services snap-in to approve a request, see the sidebar "Approving Pending Requests." If the CA has issued the certificate, you'll see a page like the one that Figure 4 shows. This page is a little confusing because it offers no clearly labeled "Download my new certificate" link. Click Download CA certificate and save the resulting file on your Exchange server. ESM expects the certificate to be stored in the Distinguished Encoding Rules (DER)­encoded format, so be sure to choose that format when you save the certificate.

After you save the certificate, you're ready to associate it with your virtual server. Return to ESM, find the virtual server to which you want to attach the certificate, and open the server's Properties dialog box. On the Access tab, click Certificate to start the IIS Certificate Wizard. This time, the wizard states that a certificate request is pending; select Process the pending request and install the certificate. On the next page, specify the path to the certificate file you just downloaded, and click Next. ESM decodes the certificate and shows you a summary screen so that you can verify that you're installing the right certificate. When you're satisfied, click Next once more, then click Finish to install your certificate. To verify that the certificate is installed, check the status of the Access tab's Communications button—this button is available only when a valid certificate is installed for the virtual server.

Enabling STARTTLS
With a certificate in place, you can start using it to protect your SMTP traffic. You have three choices: You can force the use of TLS for all outbound mail; you can use TLS for mail to selected domains; or you can force other servers that contact you to use TLS. These options are independent of one another, so you can use them individually or together.

To turn on TLS for all outbound mail on a selected SMTP virtual server, go to the Delivery tab on the SMTP virtual server's Properties page. Click the Outbound Security button to display the Outbound Security dialog box, which Figure 5 shows. Select the TLS encryption check box and click OK. However, be aware that this option effectively restricts Exchange to communicating only with hosts that support TLS. This restriction has the unwelcome side effect of making email messages sent to other hosts sit in delivery queues forever, or until the retry interval expires and the messages are returned with a nondelivery report (NDR).

A better approach is to use SMTP connectors to tailor the use of TLS to specific domains. You're almost certainly better off restricting your outbound TLS use to domains that you know support TLS. (To find out whether a domain supports TLS, telnet to port 25 on the domain's mail server and run the SMTP EHLO command from a command prompt. If the response lists STARTTLS, the domain supports TLS.) The easiest way to tailor the use of TLS is to create SMTP connectors for domains that support TLS, which you do from within ESM's Routing Groups container. When you create the connector (or modify the properties of an existing container), the two crucial steps are to ensure that you've specified the correct domains in the Address Space tab and to ensure that you've turned on TLS on the Advanced tab (click the Advanced tab, click Outbound Security, and select the TLS encryption check box).

Turning on inbound TLS is almost as easy. Go to the Access tab on the virtual server's Properties page, then click Communication in the Secure Communication command group. In the Security dialog box, which Figure 6 shows, you can turn on TLS at its default 40-bit strength or specify the full (and highly recommended) 128-bit version. Be aware that selecting the Require secure channel check box does exactly that: Servers that don't support TLS, or can't successfully negotiate a set of TLS parameters with your server won't be able to deliver mail to you. Be careful about selecting Require secure channel; if the volume of inbound mail subsequently drops, you might need to consider whether improved privacy is worth losing messages.

Troubleshooting TLS
After you turn on outbound TLS, keep an eye on your server queues. Some systems (primarily those running variants of UNIX Sendmail) will reject TLS connections from systems that use self-signed certificates, whereas others will accept STARTTLS, then won't complete the security negotiation for some other reason. If your server can't successfully negotiate a TLS connection and if it or the other system requires TLS, mail won't flow to that system.

If you see mail backing up to a particular destination system after you turn on TLS, you'll need to find out whether TLS failure is the cause. If it is, you'll need to determine why TLS is failing. Check the SMTP logs and the System and Application event logs for clues. If that doesn't help, contact the remote system's administrator.

If you still can't resolve the problem, you might be able to work around it. Although the Exchange SMTP engine doesn't support domain-specific TLS settings, you can create another SMTP virtual server that doesn't use TLS, then pair it with an SMTP connector. Set the connector's properties to include the domains that you're having trouble communicating with. Because Exchange always prefers a more specific route to a less specific one, mail bound for the specified destinations will be sent over that connector, and your mail will flow again.