Use IPsec to Encrypt Data

 Executive Summary:

Although data encryption may be the most well known (and most often used) feature of IPsec, its authentication and other features can play an important part in your defense-in-depth strategy for protecting valuable business data. IPsec often causes fear in the hearts of systems administrators, as there are many aspects of the protocol that need to be understood before it can be deployed. However, it can be relatively simple to work with if you understand the basics. Let's take a brief look at IPsec features and terminology, then I'll walk you through how to configure IPsec in a simple scenario involving two servers on a network.

What Is IPsec and What Can It Do?
IPsec is an Open Systems Interconnection (OSI) Reference Model Layer 3 (network layer) security protocol that provides strong encryption, authentication, and integrity for data that travels along the wire on parts of your network that might be accessible to untrusted hosts. IPsec has four main functions:

Digitally signs IP packets. Digitally signing IP packets ensures that data is being received from a trusted source. It can be used to help prevent man-in-the-middle attacks, and it assures you that machine X is in fact the trusted server you installed on your network.

Ensures that the data hasn’t been modified during transmission. You know that data is being received, but how can you be sure it hasn’t been modified in transit? IPsec makes sure that the data can't be modified, helping to ensure its integrity.

Provides anti-replay protection. Another security function of IPsec is its anti-replay protection. This ensures that the sequence of unicast IP packets sent between two machines remains unchanged.

Encrypts data. IPsec ensures that your business data remains confidential and can't be captured by using a network monitoring tool when it travels along the wire. The encrypting data function is an optional function: IPsec can authenticate IP packets by using digital signatures, leaving the data unencrypted if required.

IPsec doesn't rely on user authentication but uses machine authentication to ensure that only trusted hosts on your network can communicate with each other. Because IPsec is a Layer 3 security protocol, it operates transparently to users and applications. Therefore, applications don't need to support IPsec to be compatible with this form of encryption/authentication. IPsec can also help (to a limited extent) prevent attacks from trusted computers, for example by configuring restrictions on particular ports and/or protocols from remote computers to your corporate intranet.

IPsec Basics
Before we look at a sample scenario for encrypting data transmission between two servers on a network, it's helpful to know some basics about IPsec. First, let's look at IPsec headers and modes; then IPsec filter lists, actions, and rules; then authentication, cryptography, security associations (SAs), and traffic monitoring.

IPsec headers. The Authentication Header (AH) provides the ability to digitally sign IP packets, ensures packets (both IP header and data) are not modified during transmission and provides anti-replay protection for unicast IP packets. No data is encrypted. The Encapsulating Security Payload (ESP) header provides the additional benefit of data encryption.

IPsec modes. IPsec can operate in two modes: transport and tunnel mode. The default mode for Windows Server 2003 is transport mode. In transport mode, IPsec never encrypts the IP header data (information such as the source and destination address), but in tunnel mode the IP header and payload are encrypted. This is accomplished by wrapping the IP header and data inside a new packet in which the endpoints of the tunnel are unencrypted in the IP header so that the packet can be routed. Tunnel mode is used when data has to traverse a public network such as the Internet. In the example I demonstrate later, I use the default transport mode.

IPsec filter lists, filter actions, and rules. A filter list is a list of IP addresses, ports, and protocols that define one or more packet filters. A filter action determines how IPsec handles packets flowing between devices defined in a filter list, for instance to permit, block, or negotiate security. A rule is a combination of one filter list and one filter action with a few other settings such as the authentication method and IPsec mode. There can be many rules in an IPsec policy.

Authentication. Internet Key Exchange (IKE) performs “mutual authentication” between two machines. It uses one of the following methods:

Cryptography. Triple Data Encryption Standard (3DES) provides the encryption, and Secure Hash Algorithm (SHA1), the data integrity (i.e., ensures that data is not changed during transmission). Both technologies are used in combination by default in Windows 2003’s implementation of IPsec. You shouldn’t need to change the cryptography options unless you’re configuring IPsec policies with other devices that require different standards.

SAs. In Main Mode (Phase 1), IKE establishes its SA by negotiating basic security requirements such as what standards and protocols are acceptable to both nodes. In Quick Mode (Phase 2), IPsec negotiates its SA for data encryption.

Monitoring IPsec traffic. If IPsec is used to encrypt data, you won’t be able to use network monitoring software to capture data for troubleshooting purposes. However, if you choose to use IPsec for authentication only, most network monitoring tools (including Microsoft’s own Network Monitor) can monitor the traffic.

Using IPsec to Encrypt Data on a Network
Let’s look at a simple scenario in which it might be desirable to encrypt data transmission between two servers on your network. For example, you might have one server (Server A) hosting an intranet Web application that communicates with another server (Server B) that hosts the application’s back-end database. All data transmitted between the Web application itself and intranet clients connecting to it via a browser can be encrypted using HTTP Secure (HTTPS), but the data traveling along the wire between the Web server and the back-end database is left exposed. Untrusted hosts on your network might be used to sniff the data or modify it in some way.

For the sake of simplicity in this example, we’ll configure data encryption between Server A and Server B only. Figure 1 shows a simple illustration of the setup. Traffic that travels to one of these two servers from any other host is not authenticated or encrypted in any way. The IPsec policy that we’ll configure prevents the data transmitted between Server A and Server B from being captured. IPsec could also be configured to provide a greater level of protection for the servers, such as ensuring that only trusted hosts could connect to them.

Configuring IPsec Through Group Policy
Let’s configure an IPsec policy on each server by using Group Policy. An IPsec policy is required at each endpoint. An SA is established based on each endpoint’s policy configuration. It’s possible to manually configure IPsec policies without using Group Policy, but central management of such configuration is always better when possible.

Step 1: Create a new Group Policy Object (GPO). Open the Group Policy Management Console (GPMC) from Administrative Tools in the Control Panel. Right-click Group Policy Objects, select New, name the policy IPsec, and press OK. Expand the Group Policy Objects node, right-click the new IPsec policy, and select Edit. The Group Policy Object Editor window will open.

Step 2: Create an IPsec Security Policy. Expand Computer Configuration, then expand Windows Settings, Security Settings. Right-click IPsec security policies on Active Directory at the bottom of the list and select Create IPsec Security Policy. Click Next on the introductory screen, and on the next screen, give the IPsec policy a name: Web App Encrypt. Clear the Active Default Response Rule check box and click Next. Ensure that Edit Properties is selected, and click Finish. Now, to create a rule, click Add on the Rules tab and click Next on the introductory screen. We want to use transport mode, so click Next on the Tunnel Endpoint screen. On the Network Type screen, include all connections and click Next.

Step 3: Create an IPsec Filter List. At this point, we need to create a new filter list because we don’t want to encrypt traffic from all IP addresses as in the generic filter list. Click Add and give the new filter list a name: Web – DB. Click Add to start the IP Filter Wizard, and click Next at the introductory screen. Enter a description such as Encrypt all traffic between Server A and Server B only. Ensure that the Mirrored box is selected and click Next. On the Source Address drop-down menu, select A Specific IP Address and enter the IP address of Server A. Click Next and enter the IP address of Server B as the destination IP address. Click Next.

On the IP Protocol Type screen, leave the default selection of Any and click Next. Clear the Edit Properties check box, and click Finish. Click OK in the IP Filter List dialog box, which Figure 2 shows. Back at the Security Rule Wizard’s IP Filter List window, you’ll now see an additional filter list. Select the Web – DB filter list and click Next.

Step 4: Create a Filter Action. To create a filter action that requires all communications to be encrypted, click Add. Click Next on the introduction screen of the IP Filter Action wizard. Name the filter action Require Security (Strict) and click Next. Select Negotiate Security and click Next. Leave the default option of Do not communicate with computers that do not support IPsec and click Next. Leave the default option of Integrity and Encryption, click Next, then Finish.

Back at the IP Filter Action wizard, select the new filter action Require Security (Strict), and click Next. Use Kerberos as the authentication method, and click Next.

Step 5: Finalizing the IPsec Policy. In the Completing the Security Rule Wizard window, clear the Edit Properties check box, and click Finish. Back at the Web App Encrypt properties dialog box, you’ll see that the Web – DB filter is now selected, which Figure 3 shows. Click OK. In the Group Policy Object Editor window, click IP Security Policies for Active Directory, and you’ll see the Web App Encrypt policy you’ve just created. The policy isn't assigned by default, so right-click the policy and select Assign. Its status should change in the window. Close the Group Policy Object Editor window and go back to GPMC. We need to assign this GPO to Server A and Server B. Create a new OU, and link the IPsec GPO to it. Move the computer accounts for servers A and B into the new OU and log on to each server as an administrator. Run gpupdate /force from the command prompt.

Step 6: Checking the policy by using IPSecmon. You should check that each server has a successful Main Mode and Quick Mode SA. If the SAs have been established successfully, you should find that traffic is flowing between servers A and B, as well as being authenticated and encrypted along the wire. Type MMC at a command prompt, and Select File, Add/Remove Snap-In. Click Add and select IP Security Monitor from the list, click Add again, then Close. Click OK on the Add/Remove Snap-In window and expand the Main Mode and Quick Mode nodes under IP Security Monitor. Highlight Security Associations under each node in turn, and on the right side of the screen, you should see information about the SA, which Figure 4 shows. Any other host (trusted or otherwise) that attempts to connect to either server won’t require an IPsec policy.

Add to Your Overall Strategy
The example given in this article can provide one level of protection as part of an overall strategy for keeping sensitive data safe. After you’re comfortable with the concepts of IPsec, you can create more complex policies to provide greater levels of protection and deploy the protocol in a variety of different scenarios.