Use IPSec to Protect Your LAN Resources

Whether you need low-, medium-, or high-security connections, IPSec policies do the job

Attacks that take place over the Internet receive a great deal of publicity, primarily because serious attacks often interrupt service to millions of users. Malicious users who prey on corporate data from inside a network receive much less fanfare but cause at least as much damage as Internet-based attacks. (For information about the frequency of internal network attacks, see the sidebar "Why You Need LAN-based Security," page 74.) Companies rarely publicize internal network abuse statistics; acknowledging such crimes is tantamount to declaring that corporate IT can't adequately secure sensitive information.

Windows 2000 provides two methods for securing access to sensitive data: tunneling protocols (which protect a TCP/IP packet by encapsulating it within another TCP/IP packet, much like placing a letter inside an envelope) and IP Security (IPSec). These methods can reduce and potentially eliminate many internal security violations, including data theft and unauthorized data modification. You can select PPTP or Layer 2 Tunneling Protocol (L2TP) as a tunneling protocol to secure client-to-client and client-to-server connections. The advantage of L2TP is that it interoperates with IPSec.

IPSec comprises an architecture and two protocols (Authentication Header—AH—and Encapsulating Security Payload—ESP) that you can employ to secure LAN and WAN connections in either a native Win2K or mixed-mode network. After you determine the requirements for a specific secure connection, you define an IPSec policy that implements your security requirements, then activate the policy on the computers at both ends of the connection. Win2K includes built-in policies to help you enforce low-, medium-, and high-security communications; these policies are sufficient for many common network security situations.

However, the built-in policies don't cover every common security scenario, so you also need to understand how to define and test a custom policy. See the sidebar "Creating and Testing a Custom Policy" for an example IPSec policy that mandates secure, encrypted communication between Win2K systems that aren't members of a Win2K domain. By the time you finish this article, you'll have the knowledge you need to implement secure IPSec connections for Win2K domain members, Win2K standalone servers, and Win2K servers in a Windows NT 4.0 domain.

How Does IPSec Work?
IPSec employs hash algorithms and public key cryptography to secure and protect network communication between two endpoints. In its most secure form, IPSec uses one key for signing and encrypting data and a second key to verify the message signature and decrypt the data. Together, signing and encryption can provide four types of packet protection: authentication, integrity, anti-replay, and—optionally—confidentiality.

First, IPSec authenticates and verifies every message's origin, which guarantees that the sending computer is a known entity. Second, IPSec guarantees data integrity, which means nobody can modify the data while it's in transit between the sending and receiving machines. Third, IPSec uniquely identifies each IP packet with a digital signature represented as a checksum, which prevents another user from reusing or replaying a message to establish a session (possibly with stolen credentials). Fourth, IPSec can optionally guarantee confidentiality by encrypting network packets, which ensures that only the person who has the key required to decrypt the message can read the message.

IPSec provides two protocols: AH and ESP. AH protocol authenticates the sender, guarantees data integrity, guarantees that no one can replay the packet, and signs each message with a digital checksum created from either a Secure Hash Algorithm (SHA) or an MD5 hash algorithm. (The hash algorithm generates the digital checksum that acts as a packet signature.) ESP protocol does everything that AH does and also encrypts data so that no one can view it during transit. ESP encrypts data with Data Encryption Standard (DES) or Triple DES (3DES) cryptography and optionally signs each message. You can combine these protocols to achieve the highest possible level of security for computer-to-computer authentication and data transfer.

When you deploy IPSec on your network, all computers and users must be known, verifiable entities. You can rest easy knowing that IPSec security features can eliminate much network abuse. Be aware, however, that when you configure a system to require ESP protocol and DES or 3DES encryption, the encryption and decryption processes introduce a 10 to 20 percent performance overhead on the secure endpoints. (For more information about how Win2K implements IPSec, see Tao Zhou, "IP Security in Windows 2000," http://www.win2000mag.com/, InstantDoc ID 7976.)

What's an IPSec Security Association?
When you activate IPSec, the endpoints must establish a secure connection called a security association (SA) before they can securely exchange data. The SA's purpose is to verify each endpoint's identity—a process called mutual endpoint authentication—and to establish session keys. The endpoints employ Internet Security Association and Key Management Protocol (ISAKMP) to exchange the information they need to generate mutually agreed-upon keys. For security reasons, IPSec never sends the keys over the network; instead, IPSec sends the data that each endpoint needs to locally create the key. After the endpoints agree on encryption, hash algorithm, and authentication methods, they create one SA for inbound connections and one SA for outbound connections.

After the endpoints establish an SA, they can securely transfer data. An SA remains valid for 1 hour after the last packet transfer. If either endpoint tries to communicate with the other after the SA expires, the endpoints must create a new SA. You can override the expiration period and specify a different SA lifetime if the default doesn't suit your implementation. For example, on a very-high-security server, you might want SAs to expire only a few minutes after receipt of the last packet.

How Does IPSec Authenticate Endpoints?
To verify the identity of an endpoint, IPSec can use three different authentication methods: Kerberos, certificate, and preshared key. Kerberos is Win2K's default authentication method for computers and users. Kerberos authentication is the right choice if you're implementing IPSec in a native Win2K domain because you always have a Win2K domain controller to perform Kerberos authentication. In a native Win2K domain, you can safely activate Win2K's built-in IPSec policies because these policies require Kerberos authentication.

Several situations exist in which you can't implement IPSec with Kerberos authentication. If you install Win2K Advanced Server as a standalone server in an NT 4.0 domain, you have no Win2K domain controller and thus no Kerberos authentication service. The same is true if you're implementing IPSec to secure remote L2TP connections to a Win2K RRAS server (remote clients typically can't access a Win2K domain controller until they establish a valid SA). In both scenarios, you must configure IPSec policies to employ certificate or preshared key authentication.

If you opt for the certificate method, you need a Certificate Authority (CA) to assign valid certificates to computers and users. To successfully authenticate a certificate-based IPSec connection, each endpoint must have a computer certificate that verifies the endpoint's identity during the initial SA negotiation. Without a computer certificate, you'll never successfully establish an IPSec or L2TP/IPSec connection.

You can install Win2K AS's Microsoft Certification Authority, install a third-party CA, or use a public CA on the Internet. You can install Microsoft's CA in enterprise mode in a Win2K domain or in standalone mode on a Win2K server in an NT 4.0 domain. I recommend that you configure an installed CA to automatically issue computer and user certificates. The alternative requires that an administrator manually approve each request—busywork we can all do without.

If you don't have a Win2K domain controller for Kerberos authentication and you don't have a CA, a preshared key is your only option for mutual endpoint authentication. Preshared key authentication requires both systems to pass the same text string during SA negotiation. Because the IPSec policy on each machine stores the string as clear text, this form of authentication is the least secure of the three. To protect the shared string from public eyes, you should restrict IPSec policy access to Administrators only on domain controllers and restrict access to System only on the computers where the assigned policy resides.

What's an IPSec Policy?
An IPSec policy defines the parameters for secure communication between the local system and other clients and servers on the network. Every policy must have at least one IPSec rule that specifies whether to use a tunnel, the type of endpoint authentication to negotiate, and—optionally—the IP protocol to which the rule applies. The rule also specifies a filter, which provides TCP/IP source and destination addresses. (For information about TCP/IP filters, see Gary C. Kessler, Rik J. Dayvie, and Craig T. Martin, "The Power of Packet Filtering," January 2000.) Win2K includes a built-in rule, the rule, in the three native policies and in any custom policies that you define.

To view or edit a policy's rules, open Microsoft Management Console (MMC) and load the IP Security Policy Management snap-in for the local computer. Select IP Security Policies on Local Machine, as Figure 1, page 76, shows. The right pane lists the policies that reside on that machine. Client (Respond Only), Secure Server (Require Security), and Server (Request Security) are default policies that Win2K provides on every system. Each entry has a description and a Policy Assigned column that tells you whether the policy is active.

To display the IPSec configuration parameters for a policy—for example, the Client (Respond Only) policy—double-click the policy, then double-click the rule to display the Edit Rule Properties dialog box. Be forewarned: Unless you're an IPSec security guru, never modify the rule; every built-in policy uses this rule, so if you break it, none of the native policies will work.

Security Methods. On the Security Methods tab, which Figure 2, page 76, shows, you can select the IPSec protocols (and associated cryptographic and hash algorithms) that the policy agent can employ when it negotiates the SA connection. The most secure protocol (i.e., 3DES) appears at the top of the list, followed by DES, then the hash algorithms (i.e., SHA-1 and MD5). Double-click any entry in this list to view or modify the cryptographic algorithm or key lifetimes.

When the IPSec policy agent negotiates an SA, the agent offers each security method in the order that appears in this list; the endpoints select the first algorithm that they agree upon. If you include all the options in both endpoints' IPSec policies, the endpoints will always find a common algorithm. But if you restrict one endpoint to one option (e.g., ESP-3DES) and the other endpoint to a different option (e.g., AH-SHA-1), the two systems will have no algorithm in common and will never successfully negotiate an SA.

Win2K's default ordering of algorithms from most to least secure produces a steady stream of event ID 542 messages in the Application log. These messages inform you that 3DES encryption is invalid if you're exporting this system to other countries. You can eliminate these annoying events by using the Security tab's Move up and Move down buttons or the up and down arrow keys to move the 3DES entries to the bottom of the list. However, remember the dangers of altering the rule.

Authentication Methods. On the Authentication Methods tab, which Figure 3, page 78, shows, you can define the method that you want IPSec to use to authenticate the endpoints. If you click Add, you'll see a screen that lets you add or change authentication methods: (i.e., Kerberos, certificate, and preshared key). You can include one, two, or all three authentication options in your policy; the endpoints will negotiate until they find a common option.

You should never change the authentication settings for the rule because Win2K automatically includes this rule in every built-in and custom policy. If you make an error, the built-in policies won't work and you might need to reinstall the OS to restore IPSec functionality.

Connection Type. On the Connection Type tab, which Figure 4, page 78, shows, you can specify the type of connection to which the rule applies. Select All network connections for a policy that applies to local and remote connections. Select Local area network (LAN) to activate a policy specifically for LAN access. Select Remote access to apply a policy to only dial-up and VPN clients.

Built-in or Custom?
To help you implement common security scenarios, Win2K includes three built-in IPSec policies: Client (Respond Only), Server (Request Security), and Secure Server (Require Security). When you assign one of these built-in policies to a Win2K system, you define the security role for that machine.

The Client policy tells a machine to communicate without IPSec but to respond with IPSec when a machine with a more secure IPSec policy requests that it do so. In other words, when you activate the Client policy, the client responds—with Kerberos authentication and encryption—only when another Win2K system requests a secure connection. Because the policy never initiates a secure connection request, a system with an active Client policy can also communicate with NT 4.0 systems that don't support IPSec.

The Server policy always requests IPSec but permits unsecured communication if a client doesn't respond with IPSec. This process is referred to as the fallback because the policy lets the server communicate in clear text (i.e., without authentication or encryption) when the client doesn't respond securely. When you assign this policy, the local computer always tries to establish a secure connection with the endpoint. Because the policy lets the server accept incoming clear text packets, you should enable this policy only for internal servers. If you enable the policy on an Internet-connected server, you expose the server to a myriad of Denial of Service (DoS) attacks.

The Secure Server policy requires IPSec (Kerberos authentication and encryption) for all communication. When you assign this policy, every incoming connection must first establish its identity with Kerberos, then negotiate encryption. If the endpoint doesn't communicate with IPSec, the local machine will refuse the connection. When you activate this policy, local services that can't communicate securely with other systems on the network will issue frequent warnings or errors in the event log. You shouldn't assign this policy on a DNS or DHCP server because clients might not be able to authenticate until after they resolve a DNS name query or receive a DHCP address. Likewise, if the machine needs to communicate with a nonsecure DNS, DHCP, proxy, or other resource server, services on the secure system will issue failure messages in the event log because they can't connect to nonsecure servers.

So how can you effectively use these built-in policies? Suppose you have a server that publishes active-lawsuit information, and you want to ensure that only known users and computers can access the data. To ensure that other users on the network can't read the data in transit, you need to encrypt all packets that flow to and from the legal server. To secure the server, you can activate the Secure Server policy. However, if you activate the Client policy on your Win2K workstations, the workstations will never be able to access the legal server because the Secure Server policy accepts only secure connections, and the Client policy never initiates secure connections. If you instead activate the Server policy on the workstations, they will request security from the legal server and the server will respond in kind. The Server fallback option also permits workstations to communicate with systems that aren't IPSec-enabled.

Assigning an IPSec Policy
When you assign an IPSec policy, you establish communication rules for TCP/IP traffic to and from one machine. Although you can define multiple IPSec policies on the same machine, only one policy at a time can be active. To enable an IPSec policy, start the IP Security Policies MMC snap-in. In the right pane, right-click the desired policy and select Assign; to disable a policy, select Un-assign. You'll see the status of the policy change in the Policy Assigned column. When you assign a policy on a machine that already has an active policy, the policy utility automatically unassigns the previous policy. To implement the changes when you enable or disable a policy or modify a policy's properties, you must stop and restart the IPSec Policy Agent. You can restart the IPSec Policy Agent from the Control Panel Services applet. To restart the agent from the command line, type

net stop policyagent

and

net start policyagent

If you change an IPSec policy that controls remote connections on a RRAS server, restart RRAS after you restart the IPSec Policy Agent. Otherwise, RRAS won't load the new or modified policy. You can restart RRAS from the Control Panel Routing and Remote Access applet, or from the command line, type

net stop remoteaccess

and

net start remoteaccess

Creating a Custom IPSec Policy
Although the built-in IPSec policies are a good starting point, they don't cover many site-specific security scenarios. The built-in policies are good templates but provide secure-connection solutions only for Win2K domain members. All three policies include the rule, which requires Kerberos authentication, but sometimes Kerberos authentication isn't an option.

Three common scenarios call for a custom policy. First, you might need a high-security policy that mandates ESP protocol and 3DES encryption to protect a server that contains confidential information (e.g., legal briefs and depositions, proprietary scientific information, human resources data). You can't use the built-in Secure Server policy because it lets endpoints negotiate DES or 3DES, and you need a custom policy that specifies 3DES only. Second, you might need a custom policy to support remote Win2K clients that connect to a Win2K RRAS server. You can't activate the built-in policies because they all specify Kerberos authentication, and Kerberos isn't available to remote clients until after they negotiate an SA. Instead, you need a custom policy that permits remote clients to authenticate with a certificate or a preshared key. Third, you might need a custom policy to establish secure communication between two Win2K servers that are members of an NT 4.0 domain. You can't use the built-in policies because you don't have a Win2K domain controller to perform Kerberos authentication. In this situation, you need a custom policy that permits the two standalone servers to authenticate with a certificate or a preshared key. For step-by-step instructions for creating and testing a custom policy for this scenario, see the sidebar "Creating and Testing a Custom Policy."

Go for the Gold
With a careful look at where sensitive and mission-critical data resides on your network, you can implement IPSec policies that protect these resources like a coat of armor. Although IPSec can't prevent stolen passwords or an unlocked CD-ROM drive, it can ensure that only known computers and verifiable users have access to LAN servers and data and that data travels safely on your network. Perhaps next year you'll be celebrating because you've reduced network abuse by 50 percent. What a goal!

Corrections to this Article: