Defense In-depth

Using IPSec effectively in Windows 2000

The IP Security (IPSec) protocol can help you defend your system from attack. Many good magazine articles and books discuss IPSec theory and what an IPSec packet looks like. (See "Related Reading" for references.) Here, however, I discuss how to use the IPSec protocol to secure your environment.

IPSec, which is built into Windows 2000, is an Internet Engineering Task Force (IETF)-ratified standard to secure IP traffic among computers. The standard derives from IETF Request for Comments (RFC) 2401, "Security Architecture for the Internet Protocol." (For a full list of the RFCs and Internet drafts, go to http://www.ietf.org/html.charters/ipsec-charter.html.) IPSec offers a way to secure packets to make them less vulnerable. These capabilities include encryption, authentication, Hash Method Authentication Codes (HMACs), and packet filtering.

IPSec policy—in fact, any policy—is a set of business requirements implemented as technical solutions. IPSec policy determines how you use IPSec to secure connections between computers. You can configure the IPSec policy either through the Microsoft Management Console (MMC) IP Security Policy Management snap-in or with ipsecpol.exe, a tool in the Microsoft Windows 2000 Server Resource Kit. You can also obtain ipsecpol.exe from the IIS 5.0 Lockdown tool at http://www.microsoft.com/technet/security. (In the scenario I describe later, I use ipsecpol.exe because it's easier to configure IP filtering with this tool than with the MMC UI.)

Why You Need IPSec
The TCP/IP protocol lacks security features, and security threats arise because of that protocol's shortcomings. The threats include

IPSec helps mitigate these threats in several ways:

IPSec uses authentication protocols to help protect your servers against security threats. These protocols are

The Preshared key mechanism is an authentication scheme in which two or more computers have a predefined secret key; possession of this key, which the administrator determines, determines authentication. You can easily set up a preshared secret (the Preshare option) and experiment with it. If you want to use the secret in production systems, be sure it's very large and complex; otherwise, an attacker might be able to guess the key and partake in what should be secured connections.

You can use the Kerberos authentication scheme for any computers that are running the Kerberos 5.0 authentication protocol and are members of a trusted domain. Remember, Kerberos supports authentication of computers and users, which are collectively referred to as principals.

The X.509 Certificate option uses X.509 certificates between computers that don't run the Kerberos 5.0 security protocol. This option is a more secure mechanism than using preshared secrets. However, you can't set differing authentication rules between two computers for the same traffic type.

You can use the ipsecpol.exe ­n option from a command shell to encrypt and check the integrity of IPSec packets. Encryption provides data privacy, which is important in reducing damage from network sniffing attacks. Integrity checking uses hash functions to verify that data hasn't been tampered with in transit from one computer to another. Refer to the resource kit Help file for the syntax of the ­n option.

HMACs
An HMAC is a mechanism for determining whether someone has tampered with a packet of data and whether that data came from a trusted party. Essentially, a packet of data is hashed with a message digest function, such as Secure Hash Algorithm-1 (SHA-1) or MD5; a secret key is also hashed, which produces a Message Authentication Code (MAC).

When another application receives the packet, that application rehashes the data and hashes the secret key the application knows to rederive the MAC. If the two MACs are the same, then you know two things: No one tampered with the data, and someone who had access to the same secret key sent the data. Note that if you use the preshared key version of IPSec authentication, the HMAC key is different from the key used during authentication. For a more thorough explanation of HMAC and a specific implementation of a MAC, see IETF RFC 2104, "HMAC: Keyed-Hashing for Message Authentication."

IPSec in Real Life
To help you better understand how IPSec can improve your company's security, I recount the lessons the Win2K team learned when we helped build two hack sites. The team set up http://www.windows2000test.com to test a prerelease version of Win2K, and eWeek (formerly PCWeek) hosted http://www.openhack.com to test a system that included Win2K, Active Directory (AD), Microsoft Exchange Server, Sun Microsystems' Solaris, Oracle, OpenBSD (OpenBSD Project), Apache (The Apache Software Foundation), and Linux software. Both sites used IPSec effectively, but in slightly different ways.

Attackers didn't compromise the Win2K computers in either hack site. I'm not saying that IPSec receives all the credit, but it certainly helped to protect the computers. Other factors that helped were using a strong administrator password (more than 15 characters long, a combination of A-Za-Z0-9, and a lot of punctuation) and disallowing unauthenticated connections from connecting to the servers through the use of null sessions.

IPSec Packet Filtering
Packet filtering determines which network packets your system allows into your network and which packets the system should discard. For example, you might determine that certain IP packets originating from a specific network are allowed, while the same packets from any other network are discarded. IPSec lets you configure such rules, but not to the same extent as a packet-filtering firewall or router.

In August 1999, the Win2K test team set up a computer on the Internet to battle-test the computer by inviting people to attack it. By subjecting the computer to such an extreme environment, we hoped to get as many bugs as possible out of the system.

We connected the Win2K test site to the Internet without the protection of a firewall. However, although the site was utterly naked, attackers emailed us to assert that we were indeed using a firewall and to accuse us of cheating. The attackers assumed that we had a firewall because when they performed port scans against the servers, the servers dropped most of their packets.

However, our weapon was IPSec packet filtering. IPSec essentially dropped everything except traffic to specific ports that we wanted open. For example, we wanted to keep port 80 for HTTP traffic, port 443 for HTTP over Secure Sockets Layer (HTTPS)/Transparent LAN service (TLS), port 445 for System Message Block (SMB—file and print) over TCP, and port 23 for Telnet. We could also have dropped Internet Control Message Protocol (ICMP) packets (e.g., ping uses ICMP packets), but we chose not to.

Listing 1 is an example of commands that we used with the ipsecpol.exe command-line tool on the Win2K test site. The commands in Listing 1 delete the policy named Web, if such a policy exists. A policy is a set of IPSec rules, and we needed to delete Web if it already existed because, in the next few lines, we created a policy called Web. We then added a series of rules to block all traffic but let TCP ports 80, 445, and 23 stay open and ICMP packets pass. IPSec blocks all other port traffic to some extent. Win2K is hard-coded to always allow traffic to ports 88 (for Kerberos authentication) and 500 (for Internet Key Exchange—IKE). However, in Win2K Service Pack 1 (SP1), Microsoft introduced a feature that lets you filter port 88. To filter port 88 and protocol 46 (Resource Reservation Protocol—RSVP), you can add the DWORD registry subkey NoDefaultExempt to HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesIPSec and set its value to 1. (Always back up the registry before making changes to it.)

You can apply IPSec policy to different NICs in a computer by explicitly using the card's IP address. For example, the command

ipsecpol -x -w REG -p |Web -r |OkSQL-n PASS -f 157.49.211.45:1433+157.49.211.44::TCP

lets data on Microsoft SQL Server port TCP/1433 pass between the local IP address (i.e., 157.49.211.44) and the address of the SQL Server computer (i.e., 157.49.211.45).

Authentication, Privacy, and Integrity
The OpenHack attack site used IPSec in a different way from the Win2K test site. The OpenHack site limited which computers could talk to each other. For example, you have three computers (A, B, and C) and a policy that lets A and B talk to C but doesn't let A and B talk to each other, as Figure 1 shows. When this policy is in effect, if an attack compromises A, B can't be easily compromised. IPSec can help set up this type of policy.

Web-staging servers can apply a similar policy to achieve security. For example, you have six Web servers (W₁ through W₆) and two staging servers (S₁ and S₂). The security policy dictates that W₁ through W₆ can communicate with S₁ and S₂, but the Web servers can't communicate with each other, as Figure 2 shows; similarly, S₁ and S₂ can't communicate with one another. When this policy is in effect, an attacker can't use a compromised Web server as a platform to attack other computers on the network.

The commands in Listing 2 create an ISPec policy called Auth, which contains a rule called KnowSecret. KnowSecret lets only two computers, which have IP addresses 172.32.112.114 and 172.32.112.115, communicate securely.

If you want to restrict communication to any computer on the subnet that knows the secret, you can use subnets rather than IP addresses. If your subnet masks are along octet boundaries, then you can use the asterisk (*) notation to create subnets as wildcards (e.g., 172.32.*.*).

Free and Effective
The Win2K team used IPSec effectively in two hostile environments: the http://www.windows2000test.com test site and the eWeek http://openhack.com test site. I heartily recommend that you investigate using this technology as an extra level of defense when securing computers in hostile environments. It's free, and it's highly effective.

Tip: IPSec doesn't replace a dedicated firewall because IPSec doesn't perform packet-level inspection to the same degree as a dedicated firewall (e.g., inspection of the IP or TCP flags or ICMP subcodes).

Tip: You can list the rules in any order because the IPSec engine orders them, putting the most specific rules first and the most general last. this example, IPSec permits ICMP traffic and TCP traffic for ports 80, 445, and 23 and discards all other traffic.