IP Security in Windows 2000

Get Ready to Secure Your IP Network with IPSEC

[Editor's Note: This article is a revised and updated version of an article that first appeared in the August 1998 issue of Windows NT Magazine.]

In early 1998, intruders tore down Windows NT systems in many sites with a Denial of Service (DoS) attack called Teardrop2. (Teardrop2 sends an NT system deliberately constructed IP fragments that form invalid packets. The NT system receiving the packets allocates kernel memory to accommodate them. If the system receives a large number of the invalid packets, it will hang and stop working.) Although Microsoft immediately responded to these attacks with a hotfix that defends NT's IP stack against Teardrop2, NT remains a favorite target of intruders. Attacks such as these could happen again, unless network managers upgrade their NT networks' Internet protocol to IP Security (IPSec).

The Internet Engineering Task Force (IETF) developed IPSec as a security protocol of the next generation IP—IPv6. IPSec is also an optional extension for the implementation of IPv4, the current-version IP. IPv4 is widespread on the Internet and in corporate networks, but its design doesn't include security provisions. IPSecprovides confidentiality and integrity to information transferred over IP networks through network-layer encryption and authentication. IPSec protects your IP network from attacks, including DoS, man-in-the-middle, and spoofing. (For more information about IPv6 and the development of IPSec, see "The Next Generation IP in Action," June 1998.)

Microsoft has built IPSec into Windows 2000 (Win2K), which will let you implement a secure NT network without having to change your existing applications and network hardware. With Win2K IPSec, you can define security policies for your entire organization, departments, groups, or individuals, and you can specify whom your NT computers can trust and talk to and what security methods those computers can use for communication. IPSec in Win2K will be an important component of your network security.

IPSec Basics
The IETF defined the IPSec protocol in Request for Comments (RFC) 2401-2412 and several Internet Drafts. IPSec protects IP traffic with two protocols: the Authentication Header (AH) protocol and the Encapsulating Security Payload (ESP) protocol.

AH integrity ensures data integrity by authenticating a packet's IP header and payload (i.e., packet content). If a system intruder alters an IP packet and replays it, AH lets the intended recipient know that the packet underwent modification during transmission. ESP confidentiality guarantees data confidentiality by encrypting IP packets so that intruders can't decode them. ESP confidentiality is mandatory in IPSec. The difference between AH integrity and ESP integrity is that ESP integrity doesn't authenticate IP headers. ESP integrity is an option in IPSec implementation, but Microsoft recommends using both ESP confidentiality and ESP integrity for high security. However, if you use Network Address Translation (NAT) to translate your private IP addresses into Internet-legitimate addresses, you can use only ESP integrity, because ESP integrity doesn't manipulate IP headers, as AH integrity does.

IPSec operates in two modes: transport and tunnel. In transport mode, AH or ESP resides in the original IP packet between the IP header and upper-layer extension header information (to learn about the content of IP headers, see the June 1998 sidebar "What's New in the IPv6 Header?"). IPSec uses transport mode to provide end-to-end security between two end systems: for example, between a Win2K workstation and a Win2K server. In tunnel mode, IPSec places an original IP packet in a new IP packet and inserts AH or ESP between the IP header of the new packet and the original IP packet. The new IP header points to the tunnel endpoint, and the original IP header specifies the packet's destination. You can use tunnel mode to set up an IPSec tunnel between two end systems, between an end system and a security gateway, or between two security gateways. A security gateway can be a tunnel server, router, firewall, or VPN device. One example of implementing tunnel mode is securing remote access to your corporate network through the Internet. When you have a tunnel server at the perimeter of your network, telecommuters must go through the tunnel server before reaching an internal system. The tunnel in this example is between the Internet and the tunnel server (i.e., between an end system and a security gateway).

IPSec uses authentication and encryption algorithms in AH and ESP to implement data integrity and confidentiality. Several authentication algorithms are in use, such as Hash Message Authentication Code (HMAC) Message Digest version 5 (MD5), and HMAC Secure Hash Algorithm (SHA). In addition, several encryption algorithms exist: for example, Data Encryption Standard (DES), DES-Cipher Block Chaining (CBC), and Triple DES (3DES). As a minimum requirement for IPSec compliance, IPSec vendors must implement HMAC MD5 and HMAC SHA for AH and ESP integrity, and DES for ESP confidentiality. In addition to the foregoing three security algorithms, Microsoft supports 3DES for ESP confidentiality in Win2K IPSec. The sidebar "5 Security Algorithms" explains HMAC MD5, HMAC SHA, DES, DES-CBC, and 3DES.

ISAKMP/Oakley
Before two systems can exchange data, they need to agree on a set of security parameters—such as a shared session key, identity authentication method, and data authentication and encryption algorithm—that is based on their IPSec policies. This set of security parameters is called Security Association (SA). After the IETF considered several security negotiation and key-management protocols, it chose Internet Security Association and Key Management Protocol (ISAKMP)/Oakley to negotiate and manage SAs in IPSec. Oakley is a keying protocol used with ISAKMP for key exchange in IPSec. Hilarie Orman, the protocol's author, said she originally named the protocol Orman Authenticated KEYing Protocol (OAKEY), then renamed it Oakley, after legendary markswoman Annie Oakley. The IETF also defined the Internet Key Exchange (IKE) for ISAKMP/Oakley implementation.

ISAKMP/Oakley consists of two phases or modes: phase I, or main mode, and phase II, or quick mode. The main mode establishes the ISAKMP/Oakley SA, and the quick mode establishes the IPSec SA. The ISAKMP/Oakley SA is bidirectional, and the IPSec SA is unidirectional. Because the IPSec SA can travel in only one direction, two IPSec SAs for each pair of communicating systems exist.

In the main mode, ISAKMP/Oakley decides which authentication method two communicating systems will use to authenticate each other. ISAKMP/Oakley supports multiple authentication methods. Win2K IPSec uses the Kerberos, certificate, and preshared key authentication methods. After two communicating systems trust each other, ISAKMP/Oakley uses the Diffie-Hellman protocol to generate and exchange a shared symmetric key, which creates a secure channel between the two systems.

After the ISAKMP/Oakley SA is in place, either communicating system can initiate the quick mode to set up the IPSec SA. In the quick mode, ISAKMP/Oakley negotiates which security algorithms the two systems will use to secure application data. Then, ISAKMP/Oakley generates a new shared symmetric key that the selected security algorithm uses to authenticate and encrypt data. This new key is different from the shared symmetric key the main mode used, but a hashing function can derive the new key from the main-mode key. To enact Perfect Forward Secrecy (PFS), which doesn't allow use of an existing key to derive a new key, ISAKMP/Oakley can use Diffie-Hellman again to generate the new key. Win2K IPSec supports PFS. However, PFS requires additional processing time.

Both ISAKMP/Oakley modes have a rekeying mechanism that provides a highly secured session through the implementation of limited key lifetime. In limited key lifetime, when a key expires, ISAKMP/Oakley automatically generates a new key. By default, Win2K IPSec sets key lifetime to 8 hours in main mode and 1 hour or 50,000KB in quick mode. You can adjust the length of Win2K IPSec key lifetimes according to your security requirements in the IP Security Policy Properties dialog box.

Win2K IP Security Policy
The core of Win2K IPSec implementation is Win2K's IP security policy, which consists mainly of a rule that controls how IPSec works in Win2K. This rule is a collection that consists of an IP filter list, filter action, authentication methods, tunnel setting, and a connection type. Let's look at each of these policy attributes.

IP filter list. An IP filter defines whom a computer can talk to. The IP filter list can contain one or more IP filters, each of which specifies allowable IP source and destination addresses, protocol, and source and destination port numbers. An allowable address can be any address, the address of a computer receiving the security policy, a specific DNS name, a specific address, or a specific subnet. Screen 1 shows an example of an IP filter specification in which any computers in the subnet 193.1.1.0 can talk to any computer that receives the security policy containing the same filter, and vice versa. You can use an IP filter to filter any protocol, such as TCP, UDP, Internet Control Message Protocol (ICMP), and Raw IP. For example, you can use TCP and port 80 to filter HTTP traffic. A security policy can contain multiple IP filters. However, two communicating computers take further security action only when both contain a matching filter.

Filter action. IPSec uses ISAKMP/Oakley to negotiate the SA for communication between two computers. In addition, you must define security methods to specify how one computer talks to another. To define these methods, you choose algorithms depending on your security requirements. For example, you might apply 3DES for ESP confidentiality and SHA for ESP integrity to achieve a high level of security. You can define several security methods and put them in an ordered list according to your preference. When you do so, the negotiation protocol will move down the list and choose the first security method that appears on the negotiation policies of each of two communicating computers. Screen 2 shows a security method list in which 3DES and SHA has the highest priority.

Authentication methods. Win2K IPSec uses one of three authentication methods for machine-to-machine authentication. (Because IPSec is a network-layer protocol, it doesn't authenticate users.) The three IPSec authentication methods are Kerberos, certificate, and preshared key. Two communicating machines must use the same authentication method to validate each other. Win2K uses Kerberos 5 as its default authentication mechanism. In Kerberos authentication, a Kerberos server maintains the secret keys of all NT computers and users in its domain. When one machine needs to authenticate another machine in the domain, the first machine uses the Kerberos server for validation. This authentication method works well for a single administrative network, such as your NT intranet. However, if your NT computer must authenticate a computer in an external network, you can use the certificate method. In the certificate method, a certificate identifies a machine. Two machines in separate domains or networks will trust each other if a Certificate Authority (CA) that they both trust signs their certificates. Finally, the preshared key authentication method requires you to create a key or password string in your negotiation policy. If two machines share this key, they will trust each other. The preshared key method is not secure, however, because discovering a key that two systems or people share is easier for a third party than discovering a private key that only one system or person holds. Shared keys must transfer from one party to another over a secure channel. If the channel isn't secure, an intruder can steal a shared key.

Tunnel setting. You might need to place an IPSec tunnel server between two machines to set up an IPSec tunnel. You would then use the tunnel server as a tunnel endpoint in the negotiation policy. For example, to let a remote computer access your corporate network through the public Internet, you can create a security policy specifying that the remote computer must traverse the corporate tunnel server from the remote computer's public Internet address to the corporate network address. The IPSec tunnel setting in a negotiation policy contains a tunnel server's IP address or DNS name.

Connection type. Win2K IPSec supports two network connection types: LAN and remote access (such as dial-up). Through the connection-type setting, you can specify whether a security policy is for LAN, remote access, or all. If a machine has a LAN-based security policy, it can talk to another machine only through a LAN adapter—not through a dial-up modem. By adopting a LAN-based connection type, you can prevent users on remote computers from dialing up your network without using the appropriate remote access policy.

Creating an IP Security Policy
Win2K IPSec provides several preconfigured security policies: For example, the secure server policy always requests security and doesn't allow unsecured communication between clients that don't trust each other. However, you might need to create your own policies according to your company's security requirements. Let's walk through the process of creating a security policy for a company's human resources (HR) server.

Suppose that the HR server contains confidential employee and company information and lets only members of the HR department and two managers from two other departments access HR information. The HR server and all HR workstations are in a dedicated subnet 193.1.1.0. The two departmental managers' workstation IP addresses are 193.1.2.25 and 193.1.3.25. The two subnet addresses for the departmental managers' workstations are 193.1.2.0 and 193.1.3.0. The HR department can access the server using any protocol, whereas the two managers can use only a Web browser to search the employee database in the HR server. Data integrity is the minimum requirement to secure the data transferred between the server and the workstations.

Figure 1 shows the HR server's security policy. To create this policy, you first generate the IP filter list. The list in Figure 1 contains three filters. Filter 1 specifies that the server will talk to any machines in the HR subnet 193.1.1.0 using any protocol. Filter 2 and Filter 3 define that the server will talk to the two manager workstations only by TCP at port 80 for HTTP access. Filter Action lists two security methods the HR server will use: DES for ESP confidentiality and MD5 for ESP integrity, and MD5 for AH only. The server policy authentication method is Kerberos. No tunnel setting for intranet traffic exists. Finally, the HR server's policy allows only the LAN connection used for IPSec.

Using similar configurations, you can define policies for all HR department workstations, the two manager workstations, and the rest of the company. Defining security policies is an important step in implementing IPSec. Write down your security policies and save them with your network documentation.

Deploying IP Security Policies
After you define IP security policies for your Win2K network, you can set up the policies in Active Directory (AD), or you can use Win2K's IP Security Policy Management, a Microsoft Management Console (MMC) snap-in, to set up policies in individual computers. To use a specific security policy that you define on a local computer, you can manually configure the computer's IPSec option in the TCP/IP property. One computer can use only one security policy at a time. Screen 3 shows the IPSec option, in which the selected policy is HR Server Security Policy. An IP security policy in AD has a higher priority than a security policy in a local computer; the AD policy always overrides the local policy when the computer receives the policy from AD.

AD is the best place to store policies because you can manage the policies in a central repository within AD. Then, from AD you can assign an IP security policy to an entire organization or organizational unit (OU). For example, suppose you need to set up an IP security policy for the HR department of ACME, which is an OU in AD. You create the security policy in AD, create a new group security policy for the HR OU, then assign the IP security policy to the HR OU from the IP Security Policies on Active Directory setting in the Group Policy window. Screen 4 shows the HR group policy with its assigned IPSec for HR security policy. When the computer in the HR organization unit boots up, the IPSec policy agent in the computer will copy the assigned security policy from AD to the local Registry. Win2K's IP Security Policy Management lets you dynamically change security policy rules and settings. If you want to renew a security policy dynamically, local computers will check AD and automatically download the modified policy within the interval time parameter that you define. If you change a security policy often, you need to increase the frequency of the interval from the default of every 3 hours to, for example, every hour.

When an application in a local computer needs to interact with data or applications in another computer, the IPSec driver built into the IP stack examines the IP filter list in the local computer's security policy to determine whether the filter will let the local computer and its application communicate with the second computer. If a filter in the first computer's security policy matches a filter in the second computer's policy, the IPSec driver will call up ISAKMP/Oakley to negotiate the SA between the two computers based on their policies, which include machine authentication, security algorithm selection, and session key generation and exchange. ISAKMP/Oakley uses UDP at port 500 for SA negotiation. When ISAKMP/Oakley establishes the SA, the two computers can exchange application data in a secure channel. This procedure is transparent to users and applications.

IPSec Is the Key
With Win2K IPSec, you can build a secure NT network and protect it from attackers without having to modify legacy applications and upgrade network equipment; AD lets you easily manage security policies in your enterprise from a central location. Together with Microsoft, many networking and security vendors have delivered or are implementing IPSec in their products so you can leverage IPSec in your heterogeneous network. In addition to heightening your intranet security, IPSec lets you safely conduct e-commerce over the Internet. IPSec is the key to your business success.