Implementing IPSec for Front-End/Back-End Communication

Secure your intraserver communication

An important reason that organizations are upgrading from Exchange Server 5.5 to Exchange 2000 Server is that Microsoft has greatly improved Outlook Web Access (OWA) in Exchange 2000. Because more organizations are using OWA, Exchange's support for front-end servers for Internet protocol clients, including OWA, has become an important feature. Front-end servers can increase Exchange 2000's scalability and performance, especially in organizations that have a large number of Internet-protocol clients.

In a front-end/back-end configuration, the Exchange 2000 front-end server runs OWA and the back-end server hosts mailboxes. A disadvantage of this configuration is that all data that travels between the front-end and back-end servers is in plaintext, even if you've enabled Secure Sockets Layer (SSL) between the client and the front-end server. Because that data could be a message to your chief financial officer (CFO) about your company's financial records or similar confidential information, you don't want intruders reading it.

You can protect the traffic between the front end and the back end by using IP Security. IPSec, which the Internet Engineering Task Force (IETF) Request for Comments (RFC) 2411 defines, lets you encrypt IP network traffic transparently for applications such as Exchange 2000 on Windows 2000. Let's look at how you can implement IPSec to secure front-end/back-end traffic.

Introducing IPSec
IPSec lets you have a secure connection between end-to-end points across LANs or WANs or over Internet connections through Layer Two Tunneling Protocol (L2TP)/IPSec tunnels. Win2K IPSec integrates the IETF IPSec into Win2K domains and Active Directory (AD). You can use Group Policies to assign and distribute IPSec policies to Win2K domain members, or you can set up IPSec manually on each machine. Internet Key Exchange (IKE) provides an on-demand security negotiation and automatic key management service.

After peer computers have authenticated each other, they generate bulk encryption to encrypt application data packets. Only the two computers know these keys, so their data is well protected against modification or interpretation by attackers who might be sniffing the network. Because these keys are symmetric, a malicious user who had enough time and resources could break these keys. IPSec prevents such intrusion by automatically changing the keys frequently. Each peer uses IKE to negotiate the type and strength of keys to use, as well as the type of security with which to protect the application traffic.

Setting up IPSec
Let's say that you have two servers: a front-end Exchange 2000 server named FE-SERVER and a back-end Exchange 2000 server named BE-SERVER. FE-SERVER connects to the Internet with one NIC and to the LAN with a separate NIC. Here's how you encrypt the traffic between the separate NIC and the LAN.

Start the IP Security Monitor tool before you set up IPSec because the tool lets you monitor security connections. Follow these steps to start and configure the IP Security Monitor:

Creating a Custom MMC Snap-in
To make setting IPSec easier, you need to create a custom Microsoft Management Console (MMC) snap-in to hold all the snap-ins you'll use later. To create the snap-in, log on to FE-SERVER as a user with administrative rights. Then, follow these steps:

Enabling Audit Policy
Next, you configure auditing so that Win2K will log an event when IPSec is involved in communication. Auditing lets you confirm that IPSec is working properly. To enable the audit policy, follow these steps:

Activating the IPSec Policy
Now, you need to activate a default IPSec policy. A default policy uses Kerberos as the initial authentication method. To activate the policy on FE-SERVER, open the MMC Computer Management snap-in you created earlier. In the left pane, navigate to IP Security Policies on Local Machine, as Figure 4 shows.

In the right pane, right-click Server (Request Security), then change the status in the Policy Assigned column from No to Yes. Repeat these steps on BE-SERVER. At this point, you have two computers acting as secure servers. FE-SERVER requests IPSec protection before any application data is sent.

Let's look at some other policy options. If you assign FE-SERVER the Client (Respond Only) policy and BE-SERVER the Server (Request Security) policy, FE-SERVER initially sends unprotected ping packets to BE-SERVER, and BE-SERVER requests security from FE-SERVER. IPSec then secures the communication. If BE-SERVER initiates the ping, FE-SERVER must secure the ping before BE-SERVER allows the ping on the network. If you assign FE-SERVER the Secure Server (Require Security) policy, FE-SERVER won't send unprotected pings or any other traffic. Rather, it requests IPSec protection before sending any application data. If both servers have Client (Respond Only) policies, no data is protected because neither side requests security.

After IPSec security associations (SAs—one in each direction) are established between FE-SERVER and BE-SERVER, the associations remain in effect for 1 hour after the last packet travels between them. After that hour, FE-SERVER cleans up the SAs and returns to the initial Client (Respond Only) state. If FE-SERVER sends unsecured packets to the same BE-SERVER again, the server reestablishes IPSec security. The easiest approach is to let FE-SERVER send unsecured packets before IPSec security is set. This approach is safe as long as the first packets FE_SERVER sends to BE-SERVER don't contain sensitive data and as long as BE-SERVER can receive unsecured, clear-text packets from the front-end server.

Restore the IP Security Monitor window, which you minimized earlier. You'll see details about the SA that your two machines are using, as well as statistics about the number of Authenticated and Confidential bytes transmitted, as Figure 5 shows.

Now, on BE-SERVER, expand Computer Management, System Tools, Event Viewer in the left MMC pane, then click Security Log. Double-click the top instance of Success Audit in the right pane. You'll see the successful establishment of an IPSec SA, similar to the output that Figure 6 shows. (Machine names and IP addresses might differ depending on your configuration.) This information shows that you've successfully configured security between your computers.

If you're running AD, a quicker way to apply these policies is to move the servers (FE-SERVER and BE-SERVER, in this case) to an organizational unit (OU) such as Exchange Servers. Then, you can apply the policy to the OU instead of to each local computer.

If you implement the Secure Server (Require Security) policy on these servers, only IPSec clients that can successfully negotiate can communicate with the secure-server computer. Also, the front-end and back-end servers can't talk to any other systems, such as DNS servers, unless you can secure that traffic with IPSec. Because many services run in the background on the server, those services will probably fail to communicate and generate event-log messages. This behavior is normal because the default Secure Server (Require Security) policy is very strict and attempts to secure almost all IP packets before letting them into the network. For production environments, you must create a custom policy that has the behavior you want according to your security requirements, network topology, and specific server application usage. "Related Reading" at http://www.exchangeadmin.com, InstantDoc ID 24184, contains resources for more information.

Secure the Connection
Using IPSec between a front-end and a back-end server lets you use this configuration with confidence. IPSec is very flexible and powerful, but to configure it properly, you need a good understanding of the protocol.

Be aware, however, that the configuration that I explain in this article is appropriate for internal network servers only. The policy that I use allows incoming clear-text, unsecured packets. If you're placing a server on the Internet, do not use this configuration because it's vulnerable to Denial of Service (DoS) attacks that take advantage of the server's ability to receive incoming unsecured packets.