Troubleshooting IP Security Problems

Unauthorized users can access your sensitive information using several methods. Depending on your level of security, intruders can log on to a computer and access the data, capture packets going across your network, or sniff packets traveling over a public network such as the Internet. To help you combat such threats, Microsoft has included a new service called IP Security (IPSec) with Windows 2000 (Win2K) that offers machine-based authentication. But before you use the service, you should know how to troubleshoot IPSec to ensure that it's working properly in your environment. In this week’s column, I will focus on some IPSec troubleshooting techniques. For more information on how IPSec works, search the Windows NT Magazine’s article archives for IPSec.

What Is IPSec?
IPSec is a new technology you can implement on Win2K computers only (Windows NT and Windows 9x computers don't support this new technology). IPSec enhances security by protecting data that travels between computers on a LAN or a WAN. With IPSec, the computers mutually authenticate each other and establish a secure communication before exchanging encrypted data. You can use IPSec between two computers running Win2K on a LAN, between two remote Win2K-based networks, or between a Win2K computer and a Win2K-based network.

Troubleshooting IPSec
After you configure IPSec on your network, you should verify that secure communication is taking place. If not, you'll have to troubleshoot the problem. One of the challenges of troubleshooting IPSec is isolating network problems from IPSec problems. If you don't separate the two, you can find yourself on a wild goose chase. To isolate network problems from IPSec issues, use TCP/IP's Ping utility to determine whether non-secure communication is taking place. If communication is not secure, you need to check your IPSec configuration.

You can't configure IPSec between machines or networks and simply assume that your data is secure. You need to ensure that you have successfully blocked unauthorized access and that your communications are secure. You can choose from several tools to monitor and troubleshoot IPSec problems. IPSec Monitor, for example, confirms that your secure communications are really secure and offers several statistics to help you fine-tune IPSec performance. IPSec Monitor can run either on the local or a remote computer. To run IPSec Monitor, go to Start, Run, and type "ipsecmon ," where is the name of the computer that you want to monitor. Another tool you can use to verify secure communications is Network Monitor. Network Monitor lets you capture network packets and analyze them. When you examine the packets, you can identify secure packets because they display an AH as part of a TCP, ICMP, or UDP packet. IPSec packets are also displayed as ISAKMP or ESP.

IPSec can fail for several reasons. If, for example, you have configured IPSec with a computer that was part of the domain but has since moved out of the domain and is using local policy, IPSec will fail. Sometimes IPSec can fail because of network problems that have nothing to do with IPSec. Network services such as DNS, WINS, and DHCP can cause some strange IPSec behavior. In such situations, you might want to use Ping to isolate IPSec issues from network problems, as I described earlier. Another thing to try is to restart the Policy Agent to force a policy update because you might have some inconsistencies between policies. IPSec could also fail if certain files are missing. In such situations you might have to reload IP.

If you find errors in the Event Viewer, you might have a problem with the IPSec Policy Agent or the Oakley service. Use the Security Audit to investigate further.

If you're unable to establish a connection, your IPSec policy settings might be incompatible. You need to confirm which IPSec policy is in effect on the receiving. The Policy Agent might be retrieving the policy from the wrong place. The IPSec Policy Agent can retrieve a policy from the local Registry, from the domain, or from the Group Policy. You need to use the Event Viewer to filter the policy agent log events to determine where the Policy Agent is getting the IPSec policy.

Another reason for unsuccessful negotiation could be because of soft security associations (SAs) that prevent the hard SAs between computers. An SA is a contract between two computers that specifies how to protect data that travels between the two machines. You can restart the IPSec Policy Agent to clear the soft SAs, and you will also force a policy download from the Group Policy or from the domain. Frankly, restarting IPSec Policy Agent can solve a lot of IPSec problems.

If you set the key lifetime values incorrectly or too low, you might notice bad security parameter index (SPI) error messages in the Event Viewer. The receiving computer uses the SPI to uniquely distinguish between multiple SAs when the receiving computer is communicating securely with several computers simultaneously. You will also encounter SPI errors if the SA has expired but the sending machine continues to send data to the receiving machine. Use the IPSec Monitor to look at the number of rekeys. If the number of rekeys is high compared to the amount of time the connections have been active, set the key lifetime values in the policy to be longer. For busy Ethernet connections, Microsoft recommends using a number of rekeys greater than 50MB and a key lifetime value of greater than 5 minutes. Modifying these parameters doesn't guarantee that you'll end the SPI errors, but it should minimize the errors.