5 Techniques for Establishing Highly Secure Systems

Registry modifications that harden a system against TCP exploits

Microsoft has documented five TCP registry modifications you can implement to reduce a Windows 2000 system's vulnerability to Denial of Service (DoS) attacks and other common exploits. These techniques are suitable for Win2K systems connected to a WAN or the Internet and for sites operating under strict security controls. Because these modifications are sophisticated and require that you manually adjust many core TCP algorithms, I recommend that you experiment with a test system before attempting any changes on production systems. Incorrect or inappropriate TCP/IP settings can adversely affect most Win2K services and applications. Because these techniques affect the low-level algorithms that Win2K uses to manage and route messages, you should connect your test system to multiple subnets with one or more routers.

TCP/IP Registry Settings
Win2K stores all information about a running system in the system's HKEY_LOCAL_MACHINESYSTEMCurrentControlSet registry subkey. This subkey contains the Services subkey, which has one subkey for every configurable Win2K service*even services that are disabled or not installed. Figure 1 shows generic TCP/IP configuration data that the system stores in the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters subkey. (By convention, configuration data for each service is in a Parameters subkey under a subkey that corresponds with the service's name.)

To accommodate multiple adapters*whether hardware network adapters or software-based Routing and Remote Access ports*the system creates a corresponding subkey for each adapter's class ID in the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersInterfaces subkey. As Figure 2 shows, when you click an adapter's class ID, the registry editor displays adapter-specific configuration data in the right pane.

The TcpipParameters subkey contains value entries that define TCP/IP behavior for all interfaces. When you configure an adapter that supports TCP/IP connectivity, the system copies many of these settings to the adapter's Interfaces subkey. When the same value entry appears in TcpipParameters and in the adapter-specific subkey, you can modify TCP/IP operation for an individual adapter by changing the adapter's value entries. Keep the general and adapter-specific settings in mind as I describe the registry modifications you can make to reduce your system's vulnerability.

If an entry I describe doesn't appear on your system under the subkey I reference, you must create the entry. When creating registry entries, make sure that you enter the correct data type and value. You can't predict how, or even whether, a system will respond if you enter an incorrect data type or value. After you adjust TCP configuration parameters, you must reboot the system to activate the changes.

1. Protect Against SYN DoS Exploits
Malicious users make heavy use of TCP SYN DoS attacks. Before you can understand how to protect your systems from such attacks, you must understand the mechanics of TCP/IP.

To initiate a TCP over IP connection, system A sends a SYN (synchronize) packet to system B. System B replies to system A by sending an acknowledgement (SYN-ACK). If system B doesn't then receive an acknowledgment of the SYN-ACK from system A, system B retransmits the SYN-ACK up to five times. After each retransmission that system A fails to acknowledge, system B increases the interval that it waits before trying again. These intervals ensure that systems can communicate across slow networks.

If system A fails to respond, system B eventually times out the connection request. The timeout process can take as long as 3 to 4 minutes because system B must complete the required number of retransmissions to system A before it closes the connection. When system B times out the connection, TCP releases the connection resources it had allocated to the incoming connection. The process of making the resources available can take another 3 to 5 minutes.

In a typical SYN attack, a malicious user runs code on system A that causes it to flood system B with many SYN requests in a short period. Each SYN request travels in an IP packet that contains the TCP/IP address of the system that ostensibly sent the packet. This address might point to the malicious user's system, but more commonly, a "spoofed" address points to another user's system or a system that doesn't exist. Regardless, system B tries to send a SYN-ACK reply to the source address for every SYN message. When system A doesn't respond, system B must keep the connection half-open and make the required number of tries before it can close the connection. You can see that a SYN attack can quickly overload a system.

Win2K systems support hundreds of concurrent TCP/IP connections. When a system is under a SYN attack, the system can commit up to half of its available TCP/IP resources to incoming connection requests. The process of allocating, managing, and recycling the TCP/IP resources eventually hangs the system.

You can use the command

netstat n p tcp

to monitor a system that you suspect is sustaining a SYN attack. The n switch directs Netstat to display addresses and port numbers in numeric form; the p switch tells Netstat to display active TCP connections only. Figure 3 shows a system with multiple TCP connections in the SYN_RECEIVED, or half-open, state. Each connection includes the source address of the system that presumably sent the SYN. If such connections remain in the SYN_RECEIVED state for more than a few minutes, a SYN attack is likely under way. (Typically, a connection's status changes to ESTABLISHED in less than a minute.)

Win2K automatically monitors three counters that track the number of active TCP/IP ports and the number of ports in the half-open state to look for potential SYN attacks. When these counters exceed predefined thresholds, Win2K assumes that a SYN DoS attack is in progress. During a SYN DoS attack, TCP queries the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters subkey to determine how to respond. The subkey's SynAttackProtect entry determines the number of retries and the retry interval that TCP uses to time out a connection request. The SynAttackProtect entry has the type REG_DWORD and the value 0, 1, or 2.

Set the value to 1 or 2 on systems that connect directly to the Internet. These settings reduce the number of SYN-ACK retransmissions and the interval between them, which reduces the time that a system waits to time out valid and malicious connection requests. For the best protection against SYN DoS attacks, set the value to 2.

Set the SynAttackProtect entry to 1 or 2 for systems directly accessible from the Internet, especially those you don't protect with a firewall. (To verify that your firewall provides SYN attack and Ping of Death protection, check with the vendor.) Remember that when you implement this deterrent, you change the way the system responds to both valid and DoS connection requests.

2. Guard Against Dead-Gateway Exploits
When you configure a system's TCP/IP settings manually, you enter an address, a subnet mask, and a default gateway. When the system receives a packet with a destination address that's not on the local subnet, IP forwards the packet to the gateway system. The gateway then routes the packet to another router or its final destination. When the gateway is operational, it always acknowledges receipt of such packets. When a system doesn't receive a response to 25 percent of the packets that it forwards to a gateway, it assumes that the gateway is nonfunctional, or "dead."

You can guard against router failure by configuring multiple default gateways on the Advanced Settings tab of the TCP/IP Properties dialog box. When you configure multiple gateways, Win2K automatically enables a dead-gateway detection feature that selects the next gateway in the list when it detects a dead gateway. This dynamic reconfiguration adds redundancy and fault tolerance to your network, but it introduces a vulnerability that a sophisticated attacker can exploit to redirect network traffic to a nonfunctional system or a system of the attacker's choice.

The registry contains two dead-gateway entries. The first, DeadGWDetectDefault in the TcpipParameters subkey, defines the default setting for dead-gateway detection. You use DeadGWDetectDefault to enable or disable dead-gateway detection for all TCP/IP interfaces. If necessary, you can use the second dead-gateway entry, EnableDeadGWDetect in the TcpipParametersInterfacesadapterclassID subkey to enable or disable dead-gateway detection for a specific adapter. When you enable dead-gateway detection, TCP directs IP to use a backup gateway if TCP retransmits a packet several times without receiving a response from the gateway. When you disable detection, you prevent TCP from forwarding packets to another gateway.

To manually enable or disable TCP dead-gateway detection for all interfaces, set DeadGWDetectDefault to 0 (disable) or 1 (enable). To enable or disable detection for an individual network adapter, add or modify DeadGWDetect in the adapter-specific (i.e., Interfacesadapterclass ID) subkey.

When you disable dead-gateway detection, you disable TCP's ability to dynamically reroute packets using alternate routers when the primary gateway dies. Preventing packets from leaving the local subnet shuts down all but local communication. For this reason, and because the dead-gateway detection exploit is difficult to implement and is much less common than SYN DoS and Ping of Death attacks, I recommend that you disable dead-gateway detection only on highly sensitive portions of your network.

3. Battle PMTU Exploits
As Table 1 shows, each physical network type, such as Ethernet or X.25, has a maximum frame size, called the Maximum Transmission Unit (MTU), that specifies the amount of data that network type can carry in one discrete unit. When messages traverse a network of different types (e.g., from Ethernet to Token Ring), the MTU on the source network type might be larger or smaller than the MTU on the destination network type.

When a 16KB network packet must cross a network with an MTU of 1500 bytes, systems on both networks exchange MTU information. The sending system then breaks (i.e., fragments) the message into multiple packets, each with a sequence number that indicates the order of the smaller packet within the original, larger message.

Packet fragmentation is a significant source of performance degradation. The sending system consumes CPU cycles and memory when it breaks the message into multiple segments, adds a sequence number, and transmits the smaller packet. The receiving system uses CPU cycles and memory as it waits for and buffers packet fragments, orders the packets based on the sequence numbers, and reassembles the frame.

Win2K includes a performance-enhancing algorithm called the Path MTU (PMTU) that identifies the smallest MTU between a message's source and destination networks and formats the message in packets of that size. The PMTU optimization algorithm thus eliminates fragmentation at multiple routers along the path that connects the source and destination networks.

The PMTU-discovery process exposes a vulnerability that an attacker can exploit. Theoretically, an attacker can program a system to respond to the PMTU-discovery process with an MTU of 68 bytes, which is the smallest packet size that TCP/IP permits. Win2K then uses this value to package and forward all network messages to a target system outside the local subnet. The same attacker can then flood the sending system with hundreds or thousands of network packets addressed to the same target. If incoming messages arrive faster than the system can break them into 68-byte segments for forwarding to the destination machine, the attack can consume all resources on all the systems that handle the message.

To avoid this exploit, you can disable the PMTU-discovery process. You control PMTU discovery with the TcpipParameters subkey's EnablePMTUDiscovery entry. The value is of type REG_DWORD and can have a value of 0 (false) or 1 (true). By default, PMTU discovery is enabled.

Judiciously applied, disabling PMTU discovery prevents malicious users from setting the MTU, and thus the PMTU, to a ridiculously small size. However, it also eliminates an important network optimization algorithm. If you implement this hardening technique on a system that's responsible for routing network messages, you restrict the packet size for all destination subnets outside the local subnet to 576 bytes, which is slightly more than one-third of the standard 1500-byte Ethernet packet.

4. Protect Against Generic DoS Attacks
A system that initiates a TCP connection with another system periodically sends a KeepAlive packet, which is simply an ACK message, to the target system to verify that an idle connection is still operational. If the target system is still online, it replies with an ACK. If the target system is no longer online, it doesn't reply, and the sending system ultimately terminates the connection. Win2K disables TCP's KeepAlive feature by default because the algorithm introduces significant overhead, especially when a server must query hundreds or thousands of potentially idle connections.

You might consider enabling the KeepAlive feature on systems that provide resources or connections to a limited number of users. Two TcpipParameters value entries—KeepAliveTime and KeepAliveInterval—control the KeepAlive probes.

When you enable a system's KeepAlive feature, that system sends an ACK to a target system after the source system's connection with the target system has been idle for the amount of time defined in the KeepAliveTime entry. KeepAliveTime defaults to 7,200,000 milliseconds (ms—2 hours). If the remote system replies to the KeepAlive but the connection is otherwise idle, the source system sends another ACK after 2 more hours.

If the source system doesn't receive a reply, it resends the ACK at the interval defined in KeepAliveInterval. The default value for KeepAliveInterval is 1000ms (1 second). The source system queries the remote system once per second, up to five times (the TcpipParameters entry TCPMaxDataRetransmissions defines the retry count). If the remote system doesn't respond after the maximum number of attempts, the source system closes the connection.

You can reduce the KeepAliveTime value from the default of 2 hours to perhaps 30 or 45 minutes to disconnect idle TCP connections and free connection resources for other users more quickly. Remember that this setting affects all TCP connections, including local connections, so when you adjust this value, you must monitor the results closely. Microsoft recommends that you set KeepAliveTime to 300,000ms (5 minutes) when defending against DoS attacks. Unless your circumstances are unusual, use the KeepAliveInterval default value of 1 second.

5. Prevent NetBT Name-Release Attacks
By design, NetBIOS over TCP/IP (NetBT) messages aren't subject to authentication, which means that any system, including a rogue system, can send valid NetBT packets to any other system on a network. The name-release exploit leverages a logic error in how Win2K manages NetBIOS name conflicts. Typically, Win2K and Windows NT 4.0 systems discover name conflicts when the machine registers its name on the network after booting. When a conflict occurs, the OS disables networking on the system attempting to register the duplicate NetBIOS name.

The logic error lets a malicious user send a name-service datagram to a system that has already successfully registered its NetBIOS name. The name-service datagram informs the system that a name conflict exists. Upon receiving the datagram, the receiving system issues a pop-up message, releases its NetBIOS name, and stops responding to NetBIOS name queries. When a system releases its registered NetBIOS name, it loses all NetBIOS functionality, which prevents you from using NetBIOS names to browse or access systems. The name-service exploit doesn't affect TCP/IP functionality when you contact systems by DNS name or TCP/IP address.

You can use the Nbstat-n command to check whether a name conflict really exists. This command displays currently registered NetBIOS names with the word "Conflict" next to any names in conflict.

Win2K stores NetBT configuration data in the HKEY_LOCAL_MACHINECurrentControlSetServicesNetBTParameters subkey. The NoNameReleaseOnDemand entry controls NetBIOS name-release behavior. Microsoft documentation states that Win2K Server Service Pack 2 (SP2) adds this entry, but the entry doesn't appear by default in the NetBTParameters subkey. You must add it.

NoNameReleaseOnDemand has the type REG_DWORD. The default value, 0, directs the system to release its name whenever it receives a name-service datagram. If you set NoNameReleaseOnDemand to 1, the system releases its NetBIOS name only when a conflict occurs during the name-registration process. If you enable this feature, you can deter or prevent name-release attacks. If you've already disabled NetBIOS over TCP/IP on the adapter that connects to the Internet, you can skip this technique. You can only implement this deterrent on Win2K SP2 and NT 4.0 systems running the SP6a security rollup.

Microsoft Security Bulletin MS00-047 (Patch Available for "NetBIOS Name Server Protocol Spoofing" Vulnerability) cautions that enabling NoNameReleaseOnDemand causes NetBT to write multiple messages with event ID 4320 to the event log. The system will log a message with this event ID each time it receives a broadcast name-release request for common group names. These requests come from systems that previously registered the group names and are releasing these names as part of the standard shutdown process. You can safely ignore these messages.

Maximum Security
The techniques I've described are most appropriate for special-purpose, highly sensitive systems. You can use the techniques I've presented to make your systems more resilient to attacks, but the methods might negatively impact network speed or functionality.

To help you modify the registry, Web Table 2 (http://www.secadministrator.com, InstantDoc ID 25027) summarizes the relevant subkeys and entry values. Remember to reboot after you make each modification to ensure that Win2K correctly implements the change. For more information, see "Related Microsoft Articles" on the Security Administrator Web site (http://www.secadministrator.com, InstantDoc ID 25027).