9 Ways to Diagnose Windows 2003 IPsec Problems

You've implemented IPsec to protect traffic on your organization's LAN, and although you've followed all the technical documents carefully, you aren't convinced that the traffic on your network is actually protected from eavesdroppers. How can you reassure yourself that IPsec is truly encrypting your computers' network traffic?

Let's take a look at how to ensure that everything's running smoothly, then—should you discover problems in the process—look at the larger picture of troubleshooting IPsec-related authentication problems. In the process, I'll show you some built-in Windows Server 2003 tools that you can use to diagnose IPsec problems if you do find that your traffic isn't encrypted. (Note, however, that the techniques I discuss in this article don't necessarily apply to special IPsec applications, such as IPsec cards and VPN-based IPsec.)

1. Checking Whether IPsec Is Working
Even if you've successfully deployed IPsec, a lot can go wrong with it. When you suffer a loss of general network connectivity on a Windows Server 2003 or Windows XP computer, the OS makes sure that you know you have a problem. Unfortunately, there's no similar method to keep you aware of IPsec status. Depending on the IPsec deployment, when you have a problem, either you'll lose all network connectivity or—more likely and more insidious— network communications will continue working but with no encryption. Imagine your shock when you find that your network traffic is completely unsecure when you believed yourself to be protected.

The quickest way to check whether IPsec is working is to use Network Monitor to capture packets to and from your computer and check whether they're encrypted. Network Monitor ships with Windows 2003, and you can install it by opening the Control Panel Add or Remove Programs applet and accessing Add/Remove Windows Components. After you install Network Monitor, you can start a packet capture by selecting Start from the Capture menu. Then, to create some network activity and generate some data, perform a common activity such as browsing a shared folder on another computer. Finally, select Stop and View from the Capture menu.

Figure 1 shows the results of two packet captures. In the left capture, performed when a computer isn't configured to use IPsec, you see a variety of protocols listed in the Protocol column. In the right capture, performed when a computer is configured to use IPsec, you see only one protocol listed: Encapsulating Security Payload (ESP). When a Windows 2003 computer is configured to use default IPsec policies, the only protocols that will be present in a packet capture will be ESP and Internet Control Message Protocol (ICMP). ICMP will be present because the default IPsec policies permit ICMP traffic. So, if you're seeing a variety of protocols when you capture network traffic, you can reasonably assume that IPsec isn't functioning properly.

Almost all IPsec problems occur because of authentication difficulties during the Internet Key Exchange (IKE) phase of authentication. When two computers attempt to form a security association (SA), they go through a process in which they authenticate each other's identity. IKE is the algorithm that negotiates the formation of the SA. Identity authentication occurs through either a preshared key, a digital certificate, or Kerberos. The default Windows 2003 IPsec policies use Kerberos. In the large majority of cases, trouble-shooting IPsec means trouble-shooting the authentication process.

2. Restarting IPsec Services
Once you're sure that IPsec isn't working, you should first try restarting IPsec services. Restarting IPsec services will clear the IKE negotiation state completely. This strategy often restores functionality when IPsec becomes nonfunctional after significant changes in policy. Two advantages of this solution are that it works without requiring a reboot of the server and it takes only a few moments to implement. You can restart IPsec services from the command prompt on a Windows 2003 computer by issuing the following Net commands:

net stop policyagent 
net start policyagent 

Now, perform the network packet capture test again, or run one of the Netsh command tests that I talk about later.

3. Diagnosing IKE Problems in the Event Log
If restarting IPsec services doesn't solve the problem, you can move on to examining the security event log. The acts of creating and deleting SAs are audited as network logon events. If you enable success and failure auditing for the Audit Logon Events policy, these events will be written to the log. When everything is working properly, the success codes that appear are 541, 542, and 543. However, this article discusses what you should do when IPsec isn't working, so in our case, be on the lookout for the failure codes that Table 1 shows. Figure 2 shows failure event 547.

One problem with the logging of SA events is that the events can quickly congest your log. If you typically audit logon events but don't want your security event log filled with IKE entries, you can disable IKE auditing by creating the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlLsaAuditDisableIKE Audits subkey and setting its value to 1.

4. Using Shared-Key Authentication
Because it relies on a shared string rather than more complex methods such as Kerberos or certificates, shared-key authentication can provide a fallback method for getting IPsec to work. Of course, if you're already using shared-key authentication and are having problems, you should verify that the pre-shared keys on each computer actually match. IPsec won't work if one of your pre-shared keys contains a typo.

If you can get IPsec working with shared-key authentication, you'll know that the Kerberos or certificate-authentication mechanisms require more investigation. To shift to sharedkey authentication from Kerberos on the default IPsec policies, perform the following steps:

These instructions assume that you haven't heavily customized your IPsec policies. In general, you should always create a new policy rather than edit an existing one. However, in this case, you're merely adding an authentication method, which you can demote later after you resolve the problem. Changing to shared-key authentication significantly simplifies the authentication process, so it's likely to get IPsec working on your network.

5. Determining Which IPsec Policy Is Active
It's possible that IPsec isn't working because incompatible policies have been assigned to computers on the network. For example, one policy might be trying to use certificates to authenticate whereas others will accept only shared-key authentication. There are two ways to determine which IPsec policy currently holds sway. The first is the IP Security Monitor. The Microsoft Management Console (MMC) IP Security Monitor snap-in, which Figure 3 shows, replaces the Windows 2000 Ipsecmon.exe utility and can determine which IPsec policy is active on the current computer.

You can get the same information from the command line by issuing the Netsh command

netsh ipsec dynamic show all | more 

which places all the IPsec information on the clipboard. You can then paste this information into Notepad, as you see in Figure 4. In the text output, you can see that a policy is applied at the local level (i.e., Client: Respond Only) and through AD (i.e., Server: Request Security). Once you have this information, you might quickly discover that the reason IPsec isn't working is that the computer either doesn't have a policy assigned or has an incompatible policy assigned. After you resolve the policy incompatibilities, IPsec will probably function correctly.

6. Investigating Kerberos Authentication Problems
Kerberos authentication relies on AD. If a Kerberos problem arises, there's likely a problem with the computer accounts involved. You might need to reset the computer account password or re-enable the computer account. If problems persist, you might need to exempt Kerberos traffic from IPsec on the domain controller (DC). To do so, issue the command

netsh ipsec dynamic set config ipsecexempt value=0 

Win2K permits Kerberos authentication traffic, but Windows 2003 doesn't. This Netsh command forces Windows 2003 to behave like Win2K.

7. Investigating Certificate Authentication Problems
If you're having trouble with certificate-based authentication, you can probably trace the problem to one of three causes: The certificate isn't installed, the certificate is expired, or the Certificate Revocation List (CRL) check isn't completing. The default certificate period for an IPsec certificate is two years, which is long enough for most organizations to have forgotten about it. Unlike some other certificate templates, the default IPsec certificate template isn't configured for auto-enrollment.

The easiest way to determine whether certificates are installed, corrupted, or expired is to open the local computer's certificate store. The MMC Certificates snap-in is also the tool that you would use to request a new IPsec certificate. When you create an MMC Certificates snap-in to examine the expiration date of the installed IPsec certificate, you need to ensure that you select the Computer account option, as Figure 5 shows. If you select the My user account or Service account option, you won't be able to locate the issued IPsec certificate.

In the long run, the best course of action is to create a new IPsec certificate template that supports auto-enrollment (a feature that's limited to Windows 2003 Enterprise Edition). IPsec certificates will then be updated automatically, and no one will have to remember that IPsec certificates expire every two years, as long as you select the autorenewal option.

When certificates are in use, IPsec checks the CRL to check whether a certificate has been revoked. IPsec allocates a certain amount of time for this check, and it's possible that this window might close before IPsec obtains a result. To disable the CRL check, issue the command

netsh ipsec dynamic set config strongcrlcheck 0 

8. Checking Whether Everyone Is Using the Client Policy
Administrators new to IPsec sometimes apply the default Client (Respond Only) policy to all the computers on the network, then wonder why nothing is being encrypted. For IPsec policies to come into force, one of the partners needs a policy that either requests or requires the use of IPsec. If all hosts have the Client (Respond Only) policy assigned, none of them will request the use of IPsec. To resolve the problem, try activating one of the server policies on your computer and running GPU Update.

9. Checking the IKE Log
If all the aforementioned strategies don't help you with your IKE problem, you might have to take a more detailed route to resolution. At this point, you'll need advanced knowledge of Internet Security Association and Key Management Protocol (ISAKMP) RFC 2408 and IKE RFC 2409. You can enable tracing for IKE negotiations by issuing the command

netsh ipsec dynamic set config ikelogging 1 

You can increase the amount of logged information by increasing the ikelogging value to as much as 7 (which logs all possible data). The IKE trace log is located at %systemroot%DebugOakley.log, and you can use a text editor to view it. Note that the log-file content is dense, and people new to IPsec will probably be quickly out of their depth when trying to read its contents.

Beware Customization
The IPsec policies that ship with Windows 2003 typically work straight out of the box. Only when you start customizing them to use certificate-based or shared-key authentication do problems begin to arise. If you do want to modify the way that IPsec authenticates, you should create a new policy rather than edit an existing one. As long as you follow that advice, you'll be able to quickly switch to a working configuration while you investigate any problems that crop up.