Securing the Exchange Server 5.5 Internet Mail Service

In Exchange Server 5.5, Microsoft has enhanced the Internet Mail Service (IMS) with some features that tighten security for Simple Mail Transfer Protocol (SMTP) messaging. In Exchange 5.5, you can prevent SMTP clients and hosts from accessing the IMS, and you can encrypt SMTP connections.

As the Internet grows and connectivity bandwidth becomes less expensive, more companies are using the Internet as a network backbone to connect geographically dispersed networks. This configuration exposes the network to security risks that are less pervasive on a private network.

Most companies now use firewalls to protect their private networks from the Internet. However, on the Internet side of firewalls, an SMTP server often exposes email, a business-critical application, because firewalls usually leave port 25 (the SMTP port) wide open to let email into the company. Although a firewall protects the network from unauthorized access, it does not protect SMTP. In addition, users send messages in clear-text form. Clear-text messages are usually acceptable for general Internet messaging; however, if companies use the Internet as a backbone for their networks, they need to encrypt their messages. In this article, I'll explain how the Exchange 5.5 IMS protects the IMS by preventing unauthorized access and by using encryption to secure message traffic.

Authentication
Authentication lets you protect access to the IMS by requiring clients and hosts to provide a user ID and password to connect to the IMS. If you configure an IMS to require an authenticated connection, the IMS denies access to any host that does not provide a valid user ID and password.

Access control isn't appropriate for an IMS that receives general messages from the Internet, because access control limits the hosts that could connect. However, you can control access to IMS servers dedicated to intracompany messaging that uses the Internet as the company backbone, and to clients (e.g., Post Office Protocol—POP—3 and Internet Message Access Protocol—IMAP—4) that use the IMS for sending messages. (For more information about SMTP and POP3, see "Related Articles in Windows NT Magazine," page 2.)

Exchange 5.5 lets you authenticate clients in two ways: with Simple Authentication and Security Layer (SASL) and Windows NT challenge/response. With SASL, a client provides a user ID and password to the host for authentication. The client can send the user ID and password in clear text or in a BASE64- encoded string. Simple network monitoring techniques can easily compromise clear-text user ID and password security by capturing and viewing packets on the network. BASE64 encoding encodes the user ID and password and, therefore, is more difficult to compromise. Clients and hosts will automatically use BASE64 encoding if you support it.

NT challenge/response uses standard NT security to authenticate connections. The client uses NT methods to encrypt the user ID and password. Only Exchange IMS servers support this authentication mechanism.

Encryption
Encryption encodes message body data during transmission between the client and hosts. In Exchange, you can use two encryption methods: Secure Sockets Layer (SSL) and NT challenge/response. (For more information about SSL and other encryption methods, see "Related Articles in Windows NT Magazine.") SSL over SMTP uses the Enhanced SMTP (ESMTP) Transport Level Security (TLS) to encrypt message body data. To use SSL with the Exchange IMS, you must install Internet Information Server (IIS) 3.0 or 4.0 on the Exchange Server, and you must generate a key pair and attach a valid server certificate to the key pair. If you're using IIS 3.0 you can generate a key pair and obtain a server certificate from a commercial Certificate Authority (e.g., VeriSign—http://www.verisign.com). If you're using IIS 4.0, you can install Certificate Server and use it to create a server certificate and certify the key pair you generate. If you're using two Exchange IMS servers to transmit SMTP messages, you can use NT challenge/response to authenticate the connection and encrypt the message body during transmission.

Let's review the steps in enabling authentication and encryption on the Exchange IMS. You secure connections to the IMS by editing the Connections tab on the IMS Properties page, as Screen 1 shows. Under Accept Connections, if you select the default, From any host (secure or non-secure), clients or hosts connecting to the IMS on TCP/IP port 25 require no authentication or encryption. The other option, Only from hosts using:, lets you select Authentication, Encryption, or Auth and Encrypt. If you select Authentication, the Exchange IMS will require clients and hosts to provide a valid NT user ID and password (i.e., a regular Domain User account) to connect to the IMS. If you select Encryption, clients will need to support the TLS ESMTP for the IMS to support the connection. If you select Auth and Encrypt, clients will have to meet both requirements. (As of July 1998, Outlook Express in IE 4.0, Outlook 97, and Outlook 98 did not yet support SSL encryption.)

You can configure this setting for each host by selecting Hosts and configuring the Authentication requirements based on the IP address of hosts connecting to the IMS. A per-host configuration can be useful if you want to use the Internet as a backbone for intracompany messaging. In this situation, you want to encrypt your company data, but you don't want to require authentication from all hosts. Requiring authentication from all hosts would prevent your IMS from receiving general SMTP messages from the Internet, because you can't provide a user ID and password for all the SMTP servers that might connect to your server. Selecting Hosts lets you enter the IP address of another IMS server in your organization and require Auth and Encrypt for that host.

After you configure inbound authentication, you must configure the IMS servers that will connect to your IMS. For example, if you want to use the Internet as a backbone, you need to configure how the IMS servers provide the security credentials for connecting to your IMS, which now requires secure connections. To provide these security credentials, go to the Security tab of the IMS Properties page, as Screen 2 shows, and click Edit.

You configure secure outbound connections by domain. For each domain that requires a secure connection, you must provide security credentials, as you see in Screen 3. In the Outbound connections box, you configure the authentication and encryption mechanisms your IMS will use to connect to the specific domain (i.e., either SASL with the optional SSL encryption or NT challenge/response authentication and encryption). For each method, you must provide a user ID and password. For NT challenge/response, you also must provide (in the Windows NT Account box) the domain name for the NT domain in the target system.

A Typical SMTP Session
Let's look at the information that passes between client and host during an SMTP session. A network-monitoring tool shows the TCP packets that transfer information, and the Exchange IMS protocol log provides other information.

The first example is a session without Authentication or Encryption enabled. Screen 4 shows a network capture of the packets sent between two Exchange IMS servers. At the right side of the highlighted area, you can see text regarding financial information. Most messages move over the Internet in a clear-text format, as this example shows.

The next example is an authenticated and encrypted session using SASL and SSL. Figure 1, page 4, displays the IMS protocol log of the authentication taking place using SASL. Notice the BASE64-encoded user ID (AUTH LOGIN dGVzdA). To ensure that the IMS encrypted the text from this message, look at the message data shown in the network capture in Screen 5. You can see the negotiation and exchange of key information between the hosts at the top of the screen and then the encrypted message data, which starts in the line that ends in 39.

Figure 2, page 4, shows NT challenge/response authentication and encryption. The text shows the encrypted authentication that takes place beween the two Exchange IMS Servers. From this example, an observer can't determine the user ID and password combination that the IMS is using to authenticate the connection. Screen 6, page 3, shows that the message data is also encrypted, ensuring secure message transport.

Peace of Mind
With authentication and encryption over SMTP, you can now use the Internet as a backbone for intracompany messaging and maintain secure message transport. The Exchange 5.5 IMS, together with IIS 4.0 and Certificate Server, give the administrator a very secure platform on which to transfer company data over the Internet while keeping peace of mind.