Addressing DoS Vulnerabilities

Last month, Microsoft published an article that documents five registry modifications you can use to reduce Windows 2000's TCP vulnerability to a variety of Denial of Service (DoS) attacks. These guidelines are appropriate for Win2K systems connected to a WAN or to the Internet and for sites that operate under strict security controls.

Because the techniques are sophisticated and require that you modify several core TCP algorithms, I recommend that you experiment with a system in a lab setting before applying any changes to production systems. Incorrect or inappropriate settings can adversely affect nearly all Win2K services and applications. To ensure that systems you modify operate successfully in a production environment, you should connect your test system to multiple subnets, preferably with one or two routers. Today, I describe two modifications—one that hardens a system against a common DoS SYN attack and one that prevents dead-gateway exploits. For information about these and other modifications you can make, see Microsoft article Q315669.

Protecting Against SYN Attacks
To understand a SYN attack, you must understand how two systems create a TCP/IP connection and how a malicious user can exploit the connection negotiation. When one system, system A, sends a SYN request to initiate a TCP/IP connection with another, system B, system B will acknowledge system A’s connection request with a SYN-acknowledgement (SYN-ACK) message. During a typical DoS attack, a malicious user runs code on system A that sends many SYN requests to system B within a short time period. Each SYN request contains a source TCP/P address that might be real or fictitious (i.e., a "spoofed" address). In all cases, system B follows the rules for a TCP/IP connection request by sending a SYN-ACK message to the source address in the message. If system B doesn't receive an acknowledgement (ACK) of the SYN-ACK from system A, system B retransmits the SYN-ACK up to five times in an attempt to verify the connection. After each transmission that system A doesn't acknowledge, system B increases the interval before trying another transmission. This behavior ensures that systems can communicate across slow networks.

If system A never responds (e.g, if the initiating system is using a spoofed address), system B eventually times out the connection request. To accommodate slow networks, system B might take as long as 3 or 4 minutes to close unresponsive connections (the time it takes to retransmit the SYN-ACK five times and wait for a reply). When system B ends the connection, TCP releases the port so that it can reuse it for a new connection. After system B closes the port, 3 to 5 minutes might pass before the recycled port becomes available. This entire process can take as long as 8 or 9 minutes for one port, so it’s easy to understand how system B can become overloaded (i.e., the DoS condition) when it receives a flood of SYN requests.

Win2K monitors potential SYN attack activity by dynamically monitoring three counters that track active TCP/IP ports, including those currently processing a SYN request (half-open ports). When these counters exceed a predefined threshold, Win2K assumes that a SYN attack is in progress. In attack mode, TCP takes evasive action based on the value of the registry entry SynAttackProtect in HKEY_LOCAL_MACHINECurrentControlSetServicesTCPIPParameters. This value entry has the format SynAttackProtect:REG_DWORD:0, 1, or 2.

The SynAttackProtect setting dynamically adjusts the number of retries that TCP makes when the system detects a possible SYN attack. The default value of 0 is adequate for a typical LAN-based system. You can harden a system by reducing the time that TCP waits for an unresponsive connection on a system connected directly to the Internet.

You can set SynAttackProtect to either 1 or 2 to reduce the number of retries when a system doesn't reply to a SYN-ACK. When you set this value to 1 (SynAttackProtect:REG_DWORD:1), TCP reduces the interval between retransmission attempts, which causes the connection to time out sooner. When you set this value to 2 (SynAttackProtect:REG_DWORD:2), TCP further decreases the interval, which causes TCP connection requests to time out even faster. A setting of 2 hardens the TCP stack against SYN attacks most effectively. TCP frees the port more quickly and spends less time retransmitting a SYN-ACK that the target system will never acknowledge. See the Microsoft article for more information on the side affects of a SYN attack.

Protecting Against a Dead Gateway Redirect
When you configure a network adapter manually, you enter a TCP/IP address, a subnet mask, and a default gateway. The TCP component is responsible for forwarding all packets not destined for the local subnet to the gateway system. The gateway is responsible for routing the packet to its final destination. When TCP forwards a packet, an operational gateway always confirms receipt of the packet. If a gateway is down, it won't send an acknowledgement. When TCP doesn't get a response for 25 percent of the packets that it forwards, TCP assumes that a gateway is non-functional (i.e., dead).

You can guard against gateway failure by configuring multiple gateways on the Advanced settings tab of TCP/IP properties. When TCP detects a dead gateway, it selects the next gateway you define in the default gateway list and instructs IP to update the default route to use the address of the next default gateway.

You can control dead gateway detection behavior for all interfaces by adding the value entry EnableDeadGWDetect to the registry's TCPIPParameters key. This entry, which has the format

EnableDeadGWDetect:REG_DWORD, can have a value of 0 (disabled) or 1 (enabled). I suspect that you can also enable dead-gateway detection on an adapter-specific basis by adding the value entry EnableDeadGWDetect to the adapter’s subkey (i.e., under the Interfaces key), but I couldn't confirm this in the Win2K or the Resource Kit documentation.

Microsoft recommends that you disable dead-gateway detection because it prevents TCP from forwarding packets to either an unintended gateway where a malicious user might be intercepting packets or to a nonfunctional gateway, which effectively stops network traffic outside the local subnet.

Nearly all the Win2K TCP/IP documentation states that TCP performs dead-gateway detection by default. However, several of my Win2K Service Pack 2 (SP2) systems provide evidence to the contrary. My multihomed VPN server and my Win2K domain controller (DC) have a TCP/IP Parameters value entry called DeadGWDetectDefault with a setting of 0, which, I suspect, disables dead-gateway detection for all network interfaces, including network adapters and RRAS ports. Likewise, the Microsoft article states that you add the EnableDeadGWDetect in the TCPIPParameters key, but the EnableDeadGWDetect value appears as a value entry in each adapter subkey under TCPIPParametersInterfaces on my systems. Every interface has dead-gateway detection disabled. From this information, I conclude that the TCPIPParameters DeadGWDetectDefault defines the default value for EnableDeadGWDetect at the adapter level. However, I suspect that you can manually enable or disable dead-gateway detection on a per-adapter basis by editing this value in each interface key.