Encrypting Internet Email with Exchange Server

Lock down an SMTP connection at the server, not at the client

Email creates a new instance of an age-old dilemma: security versus convenience. We face this dichotomy in many aspects of our daily lives. For example, when we leave for work in the morning, most of us lock our homes behind us. Although we'd prefer not to deal with keys and locks, the simple fact is that our possessions would eventually disappear if we left them continuously unprotected. So we put locks on our doors and sacrifice a little bit of convenience for the sake of security.

The reverse—sacrificing security for the sake of convenience—also occurs, specifically in some Microsoft products. Password caching in Windows and Microsoft Internet Explorer (IE) is a prime example. Although many network administrators don't approve of password caching, users' workstations can store passwords so that users don't need to remember them.

Email represents the ultimate in communication convenience coupled with a complete lack of security. The analogy that compares the security of an email message to that of a postcard is absolutely accurate. When SMTP was developed—back when the Internet was a much more trusting environment—security wasn't a major concern. As a result, users today transmit confidential information across the Internet without realizing just how vulnerable that data is.

Suppose two pharmaceutical companies join forces to develop a revolutionary new drug. Scientists at the two companies need to share data and documents concerning their research, and they want to do this via email because doing so is convenient. But what if the scientific documents, as they travel across the Internet in an email message, fall into the hands of a competitor? If that competitor then beats the collaborating companies to a patent on a new drug, the potential lost revenue could amount to billions of dollars.

Obviously, network administrators at the two companies need to secure the email traffic that travels between the scientists. How do the administrators best achieve such security? They can rely on users to encrypt email messages, or they can configure email servers to encrypt messages.

Client or Server Encryption
Microsoft Outlook email clients (Outlook 2000 and Outlook Express) have supported encryption and authentication mechanisms for some time. When you properly configure Outlook, users can simply click a button to encrypt (and optionally sign) a message before sending it. Although this security solution is functional, it has two minor flaws. The first is that the message recipient must have the appropriately configured software (and education) to decrypt the message. The second (and more significant) flaw is that the encryption process depends on an unreliable link: end users, who must remember to encrypt their email. Just as we all occasionally forget to lock our houses and cars, users occasionally forget to encrypt confidential messages.

If you're running Microsoft Exchange Server 5.5 or later, server-based encryption is a better alternative. Exchange Server can encrypt SMTP conversations between mail servers across the Internet. This feature is surprisingly easy to set up and ensures the encryption of all messages sent to a designated domain.

To configure server-based encryption on the sending end, you define specific Internet email domains or subdomains in Exchange Server, then instruct the server to encrypt messages headed to accounts in those domains. For example, I could direct my server to encrypt all messages to the domain ntsol.com before transmitting them.

The administrator of the Exchange server on the receiving end defines a special user account for the encrypted messages. If I've configured my server to encrypt all email going to the domain ntsol.com, I need to ask the administrator of the ntsol.com domain to create an account for my Exchange server to use. Let's say the administrator at the remote site creates an account named DOUG.

At this point, I have simple one-sided encryption, in which all the email I send to ntsol.com is encrypted and all the email that ntsol.com sends me is in plaintext. However, I want encryption to work both ways, so I need to set up a user account in my domain. Then, the administrator at ntsol.com can program his or her mail server to encrypt email that it sends to my domain.

Configuring Encryption
To set up a domain and account for encryption, you must modify settings on the Exchange servers that handle your Internet email. Start Microsoft Exchange Administrator, open the Internet Mail Service Properties page, and click the Security tab, which Figure 1, page 88, shows. By default, the system secures no domains with encryption. Click Add to open the Edit E-Mail Domain security information dialog box, which Figure 2, page 88,shows.

Type the name of the domain you want to secure (in my case, ntsol.com), then select the Windows NT challenge/response authentication and encryption option. Next, you must define the account credentials your Exchange server will use when communicating with the other Exchange server. In effect, your server must log on to the other server. To define the account your server will use, click Change. In the dialog box that opens, type the NT domain name, account name, and password to use. (I typed INTELLINET for the NT domain and DOUG for the account.) Remember that your system will use this domain name and these user account credentials when connecting to the remote system—therefore, for encryption to work, the remote system administrator must set up the user account you define in this dialog box. After you enter the appropriate authentication credentials for your Exchange server to use, you'll see the NT domain and account names in the Edit E-Mail Domain security information dialog box's Windows NT Account field.

Now, whenever my Exchange server sees an outbound message headed for any email address ending with @ntsol.com, it will automatically encrypt the message. The address might be [email protected] or [email protected]—Exchange Server will encrypt all @ntsol.com addresses, regardless of content. An administrator doesn't need to add new addresses for encryption or install software on workstations.

NT Challenge/Response authentication enables NT LAN Manager (NTLM) encryption for an SMTP session. The strength of the encryption depends on your OS installation. If you have the 128-bit (US-only) version of NT at both ends of the connection, you'll get 128-bit encryption. If you have the 40-bit (exportable) version of NT at both ends of the connection, you'll get 40-bit encryption. If you have a different version at each end, NT will probably negotiate down to the 40-bit level because the configuration dialog box doesn't offer an option to "require" strong encryption.

Server encryption is a great feature if you need to share confidential information with business partners who run Exchange Server. You can secure all the email traveling between your systems in just a few minutes by enabling this option.

Always a Skeptic
Given how easy server encryption was to set up, I was a bit skeptical about whether it actually worked. So I broke out the ol' packet sniffer and captured some of my SMTP traffic to see just exactly what was going on.

I started my test by deleting my new email domain ntsol.com so that all messages from my Exchange server would travel across the Internet in clear text. (To delete ntsol.com, I selected the domain from the list on the Internet Mail Service Properties page's Security tab and clicked Remove.) Then I sent a message to Windows 2000 Magazine columnist Sean Daily (sean@
ntsol.com) and captured it with my sniffer. Figure 3 shows what you might expect: the SMTP session details and the easily readable contents of the message ("Hi Sean ­ this is a test, unencrypted message").

I then redefined the ntsol.com domain and sent a new message. As Figure 4 shows, the message text is garbage to the naked eye. If you look at the top pane of the capture window, you'll see that at packet number 260, my server sent an Auth command followed by NTLM (the type of authentication) and garbage characters. The garbage characters represent the email domain I just redefined and the account name and password. After you set up the authentication session, the remainder of the message—the header fields and the message body—is encrypted.

If, for some reason, the authentication process between two Exchange servers breaks down and the servers can't establish an encryption session (e.g., a new administrator deletes the account that the Exchange servers use to authenticate each other), the message sender will receive an error message such as the one that Figure 5 shows. The message isn't sent in clear text. The error message's "535 ntlm authentication failed" indicator tells you that the Exchange servers can't set up an authentication session and deliver the message until an administrator corrects the problem.

Encryption with Non-Exchange Servers
What if you need to exchange secure email with a non-Exchange server? You can accomplish this task if the non-Exchange server supports SMTP with Transport Layer Security (TLS)—aka Secure Sockets Layer (SSL)—through Enhanced SMTP's Starttls command. The Internet Engineering Task Force (IETF) Request for Comments (RFC) 2487 covers SMTP with TLS, describing how servers can set up encryption sessions with one another for securing email. The easiest way to find out whether a non-Exchange email server supports SMTP with TLS is to start a Telnet session to the non-Exchange server on port 25 (the default port for SMTP) and simulate an SMTP-with-TLS session. To do this, type the following command in the Telnet session:

ehlo 

Wait for the responses from the server, then enter

starttls

Figure 6 shows such an exchange. As you can see by the response "Ready to start TLS," this non-Exchange server supports TLS. To enable TLS on an Exchange server, you use the Edit E-Mail Domain security information dialog box. Type an email domain name, select SASL/SSL security, and select the SSL encryption check box. However, to activate SSL encryption on an Exchange server, you need a server certificate. For more information about requesting an SSL certificate and enabling it for your Exchange server, see the Microsoft article "XFOR: Connecting Internet Mail Service (IMS) to IMS with Simple Authentication and Security Layer (SASL)" (http://support.microsoft.com/support/kb/articles/q174/7/54.asp).

Complicating Factors
A firewall or SMTP content-scanning system can throw a monkey wrench into a server-based encryption plan. Such devices are frequently used to secure an organization's Internet connections or to scan inbound and outbound email, and they could intercept or modify SMTP sessions.

If you have one of these devices—each of which functions a bit differently—you'll need to test your environment to see whether NTLM encryption will still work for you. One possible solution if it doesn't—and if your security policies allow it—is to make an alternative inbound/outbound path to your Exchange server (perhaps through a second NIC) that goes around your firewall or content-scanning system and responds to a different host name (e.g., securemail.domain-a.com). Then, you can enable NTLM on the alternative interface and instruct users on the remote Exchange server to address sensitive messages to the appropriate users at securemail.domain-a.com. The remote SMTP server will establish a direct connection to your server's alternative interface and encrypt the SMTP session through NTLM.

This solution isn't perfect because it relies on users on one side of the encrypted connection to remember to send messages to the correct address. But it might be better than no encryption at all. Of course, taking full advantage of NTLM email encryption between servers is best. This Exchange Server feature can provide your organization with a lot more security at little additional cost or maintenance complexity.