Protect Private Ports with IPSec

Protect ports for the services your Web server requires

In "Close the Doors to Your Web Server," February 2002, InstantDoc ID 23573, I discussed the importance of disabling unneeded services to close down the related ports and prevent them from being potential targets for attack. But how do you protect ports related to the services that your Web server requires? I find it useful to divide the open ports on a Web server into two categories: ports that the general public accesses and ports that only computers under your control access. For example, the general public usually needs access to ports 80 (HTTP) and 443 (HTTP over Secure Sockets Layer—HTTPS). To protect against attacks that target public ports, you must secure both the internal application and the OS.

However, you probably have other ports open on your server—to support such activities as content maintenance, administration, and updates from databases in your network or a business partner's network. For example, if you administer an IIS Web server from a remote computer through that computer's Web browser, a much more limited set of computers should have access to the port assigned to IIS's Administration Web Site. Similarly, if you use Windows 2000 Server Terminal Services, a limited set of computers will have access to port 3389. If you have an FTP server enabled on your Web server to maintain content or support file transfers to and from a database, a similarly limited set of computers will have access to ports 20 and 21. You might also want to protect access to the Remote Registry (port 135) and Telnet (port 23) services.

Not only are these private ports vulnerable doorways into your Web server, but securing and keeping all the services that relate to these ports up-to-date with software updates and service packs is difficult. For these private ports, Win2K's IP Security (IPSec) functionality can come to the rescue. You can easily create an IPSec policy to require that an authenticated and encrypted IPSec connection be established before any computer can connect to these more private ports.

How an IPSec Policy Can Help
An IPSec policy lets you use strong authentication to limit which computers can connect to a port. An IPSec policy also ensures that IPSec encrypts all subsequent communications and performs integrity checking. These IPSec operations are transparent to the services and client applications that use the port; however, you're effectively wrapping a service such as Terminal Services in a new level of security. You've added protection in case you inadvertently configure Terminal Services insecurely or an intruder discovers a vulnerability in your version of Terminal Services. You're even protected against Denial of Service (DoS) attacks related to Terminal Services or any other private service you protect with IPSec.

You get the additional protection because IPSec prevents an untrusted computer from connecting to the relevant port in the first place. To connect to an IPSec-protected port, the computer must successfully authenticate as a trusted computer by using one of three authentication methods (Kerberos, preshared secret key, or certificate-based) you specify. Without IPSec, you can limit connections from untrusted computers only by filtering on a source IP address. (Source filtering is vulnerable to spoofing, whereas IPSec's filtering isn't. Spoofing involves an attacker sending packets in which the source address has been substituted with a fake address to fool the target computer into thinking it's communicating with a trusted computer. ) In addition to authenticating each packet, IPSec also performs integrity checking through which it detects modified packets and rejects them, and session hijacking.

The only requirement for using IPSec to protect your private ports is that the computers that access these ports must support IPSec. You can enable IPSec support more easily between computers that run Windows XP and Win2K. For purposes of this article, I assume that the computers accessing private ports on your Web server run Win2K. You need to configure complementary IPSec policies on these computers (i.e., if the server requires a certain level of security and authentication, the client must be configured to acquiesce) to let them communicate with your Web server. In this article, I show you how to plan your IPSec implementation; in a subsequent article, I'll show you how to configure IPSec for a sample Web server.

Assigning an IPSec Policy
For each Win2K computer, you can assign (i.e., activate) one IPSec policy. A policy is a set of rules, as the sample policy in Figure 1 shows. A rule catches packets that match specified filter criteria, then applies one of three actions to those packets: block, permit, or negotiate security. The three main parts of a rule are its IP filter list, its designated filter actions, and its authentication methods.

First, the filter list contains filters that specify any combination of protocols (e.g., TCP, UDP), source and destination IP addresses, and port numbers in much the same way that firewalls filter for specified elements. After the filter list catches a packet, Win2K moves to the second part of the rule and applies the filter action. You can specify whether Win2K permits the packet, blocks the packet, or negotiates a secure IPSec connection. Negotiation lets you require that the connection be subject to IPSec's Authentication Header (AH) mode or Encapsulating Security Payload (ESP) mode. In AH mode connections, IPSec authenticates and performs integrity checking for each packet. In ESP mode connections, IPSec also encrypts everything in the packet except the destination address.

The third part of a rule contains the rule's authentication methods. For permit or block actions, you don't need to specify an authentication method. However, if your rule's action is to negotiate security, you'll need to ensure that at least one of the three authentication methods is configured. (If more than one authentication method is configured, the two computers compare the methods for which each one is configured until they find one they have in common.) The available authentication methods are Kerberos, preshared secret key, and certificate-based authentication.

Unlike router or firewall tables, the sequence of rules within a policy or filters within a filter list isn't important. You can specify a blocking or negotiate-security rule with a broad filter list, then open ports with a permit rule whose filter list specifies just the public ports. Win2K automatically applies what it considers the most specific rule to each packet. Although you might think that this approach could lead to some ambiguity, so far Win2K has always interpreted the policies I've created as I intended.

Protecting Your Private Ports
The simplest way to protect your Web server's private ports is to create a broad rule that requires negotiated security for all connections to the Web server from any source address. Then, create a more specific rule that permits connections to public ports such as 80 and 443 without any further IPSec involvement, as Figure 2, page 15, shows. For your broad rule that requires negotiated security, you'll need to choose an AH or ESP mode.

If you need to simply ensure that only trusted computers access the port and you don't care whether others can see the data in the packets, you can improve performance by using only AH. Some applications such as Terminal Services already encrypt some or all of the data they transmit. To ensure maximum confidentiality, use ESP. (When you use ESP, even the port numbers are encrypted.) Incidentally, my testing hasn't revealed that implementing IPSec imposes a significant performance penalty. (I transferred a file of at least 2GB without IPSec in about 45 minutes. With IPSec in ESP mode, the file transfer took only 3 minutes longer.)

With the flexibility of IPSec, you can define rules that help you stop worrying about attackers gaining access to your Web server through the private ports you use to administer or update your server. If you use the strategy I've described—of creating a broad rule that requires IPSec security for all ports—don't forget to create an exception rule to allow unsecured communication to public ports such as 80 and 443. Otherwise, you'll lock the public out of your Web server.

I discussed the overall pieces involved in limiting access to private ports on your Web server, but how does IPSec know it's communicating with a trusted computer? You'll need to determine which authentication methods to use.

Kerberos is the easiest authentication method to designate in your IPSec policy because you don't need to set up any additional key exchange or certificate requests. However, Kerberos isn't an option for computers that aren't part of trusted domains, which leaves preshared secret key or certificate-based authentication.

Next month, I'll consider the pros and cons of each authentication method for Web servers. I'll also set up a sample IIS server to show you how to use IPSec protection.