Packet Filtering and Windows

Of all the recommendations for securing a server, probably the most pervasive advice is to install some kind of packet filter between the server and any untrusted network. Many security strategies are based on the simple concept of reducing the attack surface. By using a packet filter to block or restrict network traffic, you can reduce the number of potential entry points that an attacker might take advantage of.

Packet filters usually come in the form of a firewall. A firewall can serve many purposes, but its primary function is to filter network packets and allow only traffic that matches specific predefined rules. However, a firewall isn't the only way to filter packets. Windows provides various built-in packet-filtering options to provide a primary or secondary layer of network protection.

Many organizations already protect crucial servers by implementing a firewall in a demilitarized zone (DMZ) between the server and the outside world. But sometimes you need additional packet-filtering capabilities on the server itself. For example, you might have a server hosted at a remote co-location facility that doesn't provide a firewall or any kind of packet filtering. In this case, it can be more convenient to use a packet filter on the server rather than install a separate firewall device.

Another reason to install an additional packet filter on individual servers is to provide multiple layers of defense. If an intruder gains access to one or more servers inside the main firewall, having a separate packet filter on the server can limit the scope of the attack.

Windows includes three options for packet filtering on a server: TCP/IP filtering, IP Security (IPSec), and—in Windows Server 2003 and Windows XP—a built-in firewall. In addition to these built-in options, many free or inexpensive host-based firewall solutions are available. However, no single packet filter is the best choice for all scenarios. Looking at the pros and cons of each of these technologies can help you decide which is right for you.

Filtering Requirements
A host-based packet filter should have little impact on server performance—the software shouldn't consume excessive system resources or slow down network traffic. The software should provide flexible rule options, be able to filter both incoming and outgoing traffic, be simple to configure, and let you easily duplicate or script a configuration.

Another important consideration is whether the packet filter you implement is stateful. A firewall works by blocking certain packets according to information in the packet header. By checking for a TCP state flag in a packet's header, the firewall can usually tell whether the packet is a new incoming connection or is a response to an outgoing request. However, some protocols throw firewalls a curveball. FTP, for example, requires two TCP connections: one to exchange commands between the client and the server and another to transfer files. In typical FTP mode, the server must initiate the file-transfer connection to the client, and the packet filter needs to be able to determine that that connection is in response to the original FTP request. The only way the packet filter can make that determination is to keep track of recent traffic. A stateful packet filter keeps a table of recent packets so that it can dynamically create new filters according to previous connections.

UDP traffic can also cause problems. UDP packet headers don't contain a state flag to show whether the packet is a new incoming connection or a response to a connection initiated by the client. For example, DNS uses UDP port 53 to look up a host name for an IP address. The response to such a request looks like a new incoming connection, and only a stateful packet filter can associate the response with the original request.

Stateful packet filtering adds intelligence to the filtering process because it's aware of previous connections. It also provides better protection against port scanning and Denial of Service (DoS) attacks that take advantage of flaws in network protocols. The drawback to stateful packet filtering is that it increases overhead. The extent of that increase depends on the nature of your network traffic.

Stateful packet filtering is especially important if you have a crucial server that also initiates a lot of outgoing client traffic. Using a stateful packet filter simplifies your firewall rules and prevents stateless packet-filtering rules from opening holes. Without stateful packet filtering, you must carefully design filtering rules to prevent such holes.

TCP/IP Filtering
Simple to configure and unambiguous, TCP/IP filtering has been around since Windows NT and is the most basic form of packet filtering available in Windows. Through the simple interface that Figure 1 shows, TCP/IP filtering lets you specify which TCP and UDP ports to permit access to on the server and lets you limit access to specific IP protocols.

TCP/IP filtering isn't stateful and works only on incoming network connections. Thus, it can't filter outgoing traffic, and it can't filter incoming traffic that responds to an outgoing request. TCP/IP filtering also doesn't properly filter Internet Control Message Protocol (ICMP) traffic. TCP/IP filtering has no logging capabilities, and its packet-filtering rules are limited to matching the destination port or protocol. TCP/IP filtering can't block or allow traffic from a specific IP address and doesn't let you specify ranges of ports. Finally, changing TCP/IP filters often requires a reboot, which might not be acceptable for crucial servers.

With all these limitations, TCP/IP filtering might seem like a poor choice for packet filtering. But it has one distinct advantage: Because it works in kernel mode, it's extremely fast. With TCP/IP filtering enabled, you'll notice little if any drop in network performance. For some applications, speed might be more important than advanced filtering capabilities. If that's the case and the server configuration is simple, TCP/IP filtering can be an acceptable solution.

IPSec
IPSec is a good alternative to TCP/IP filtering, and systems administrators frequently use it as a packet filter on standalone Windows servers. Designed to secure IP communications, IPSec can encrypt and tunnel network traffic, but it also offers flexible packet-filtering options and is often used for that feature alone. IPSec is difficult to set up, but after you configure it, you can easily use scripting or Group Policy to distribute an established rule set.

IPSec's enhanced rule capabilities let you filter by specific addresses, protocols, ports, or any combination of these. Most important is that IPSec lets you filter both incoming and outgoing traffic. By allowing only certain outgoing traffic, you can greatly reduce the impact of worms and Trojan horses. For example, if you disallow outgoing traffic that's directed to TCP port 25, no one can send email from your server and you can prevent someone from using a Trojan horse or a worm to make your server distribute spam.

IPSec does have its disadvantages. As I mentioned, configuration is challenging. In addition, the terminology is sometimes confusing, and many people have trouble understanding the differences between rules, rule sets, filters, and filter sets. Even firewall experts get confused because, unlike most other packet filters, which order rules sequentially, IPSec orders rules from most specific to most vague.

Another problem with IPSec is that it slows performance. I've seen IPSec packet filtering reduce network bandwidth by 10 to 15 percent even when encryption and tunneling aren't being used. The performance penalty isn't always a problem, but if you're already pushing your server to its limits or speed is crucial for your application, IPSec might not be the best solution. Because every environment is different, you should run tests to determine exactly how much IPSec will affect your server.

IPSec isn't stateful, but because you can use rules to specify the source and destination ports, IPSec is more flexible than TCP/IP filtering. For example, client traffic typically originates on an ephemeral port (usually a port number from 1024­4999). With IPSec, you can build flexible rules that permit only traffic that originates from ports in this range, so attackers can't modify their source port to bypass your filtering rules. Specifying the source ports also limits exposure to some types of DoS attacks.

IPSec lets you log filtered connections (but not packet contents). However, IPSec stores the logs in the Windows event log, which makes browsing blocked packets inconvenient because you have to double-click each event to view log details. Furthermore, if you have a busy server, IPSec logging can quickly fill up the server's event logs. To handle IPSec logs, you might want to use Microsoft's Log Parser tool, which lets you easily extract and save IPSec events in any log format you want. You can download Log Parser from the Microsoft Download Center at http://www.microsoft.com/downloads/search.aspx?displaylang=en. For information about using the tool, see "LogParser," May 2004, InstantDoc ID 42174, and "Event-Log Fields," May 2004, InstantDoc ID 42175.

If you can tolerate slower performance, IPSec is a good solution for a standalone server—especially one in a co-location center where you can't install a separate firewall. IPSec's rule-set capabilities are flexible enough for most scenarios, and its logging capabilities are fair.

The Built-In Firewall
If you run Windows 2003 or XP, you might want to consider using the OS's built-in firewall, which comes in two versions. The original version, called Internet Connection Firewall (ICF), ships with both OSs, and a greatly enhanced version renamed Windows Firewall will be available with Windows 2003 Service Pack 1 (SP1) and XP SP2. The two versions are so different that we really must consider them separately.

ICF. ICF is a simple stateful firewall that by default blocks unsolicited incoming traffic. You can make exceptions to that behavior by permitting traffic on specific TCP or UDP ports to allow access to server applications. ICF doesn't block outgoing traffic and has limited filtering capabilities for incoming traffic. It lets you open ports for server applications, but those ports are then open to everyone—you can't limit access to those ports by source IP address.

Although ICF doesn't impact performance as much as IPSec does, it does have an effect. For most applications, you should expect a drop of 5 to 10 percent in network performance. ICF stores its logs in a plain-text file, and you can choose to log dropped or successful packets.

With its limited filtering capabilities and inability to block outgoing traffic, ICF might not be the best solution for a server. Furthermore, ICF requires starting additional services, namely the Remote Access Connection Manager and Telephony services.

Windows Firewall. Windows Firewall is Microsoft's response to the ever-growing threat of worms and other Internet-based attacks. Microsoft added many security features to Windows Firewall that don't appear in ICF, including

These new features make Windows Firewall a major improvement over other built-in packet-filtering options. Windows Firewall's effect on performance is about the same as ICF's, but Windows Firewall has much more flexibility—it even lets you specify which applications can open listening ports.

Like ICF, Windows Firewall can't filter outgoing traffic. However, it does provide stateful packet filtering, making it much easier to configure than stateless technologies. Windows Firewall also lets you block all incoming traffic, then make exceptions for specific applications, as Figure 2 shows. The enhanced Netsh support makes scripting easy, and you can enforce firewall settings through Group Policy. If optimum performance isn't critical and you don't need fine control over firewall rules, Windows Firewall might be a suitable option.

Other Solutions
In addition to the built-in solutions, dozens of third-party firewalls are available, many of which fall into the category of personal firewalls. These products generally have powerful features and flexible rule capabilities but occasionally fall short in performance and stability, especially under heavy traffic. The biggest problem with many of these products is that they aren't well designed for use on a server. Many personal firewalls use pop-up message boxes and tray icon notifications that require user interaction and make the firewall impossible to manage remotely, but some vendors are catching on and providing for remote configuration and management. Because there are many personal firewall products and a great deal of variation among them, make sure you carefully consider the specific requirements of your environment before you decide to use one.

Yet another solution is to use a NIC that has built-in firewall features. Some of these hardware solutions provide full-featured packet filtering, whereas others simply speed up IPSec by handling that processing in the NIC hardware. NIC-based firewalls and IPSec offloading provide excellent performance, but features and quality vary from product to product.

Something Is Better Than Nothing

No one packet-filtering option is perfect for all situations. However, they can all supplement other solutions, and any type of packet filtering is certainly better than nothing. TCP/IP filtering works well for simple configurations where performance is crucial, but it provides limited filtering capabilities. IPSec offers flexible configuration options and can protect both incoming and outgoing traffic, but at the expense of performance. Both ICF and Windows Firewall provide stateful packet filtering but can't protect outgoing traffic. Table 1 summarizes the features of the built-in options.

Depending on the server role, one or more of these solutions might be sufficient. Just remember that the objective is to reduce exposure. The more holes you plug, the less likely your server is to be vulnerable to current and future attacks.