Access Denied: Troubleshooting IPSec

How can I make sure that IP Security (IPSec) works correctly when I first set it up? Also, how can I confirm that our IPSec policies are active and securing communications?

You can use the IP Security Monitor tool (ipsecmon.exe), which resides in %systemroot%system32, to display the current status of IPSec on the local computer. Windows 2000 refers to a negotiated IPSec session between the local computer and another computer as a security association. IP Security Monitor displays all active security associations in the Security Associations section at the top of the tool's GUI, which Figure 1 shows.

To see the tool in action, configure the local computer and a remote system with an appropriate IPSec policy, then assign the policy. Next, initiate a communication between the two computers that the policy will intercept and secure. For example, configure the local computer to require security for incoming Win2K Server Terminal Services connections and configure the remote system to acquiesce to any request for secure communications. (For more information about how to set up IPSec and Terminal Services, see "Terminal Services, Part 4," http://www.secadministrator.com, InstantDoc ID 20288.) On the remote computer, open a Terminal Services session to the local computer. Open IP Security Monitor on the local computer. The tool will display an active security association that specifies the level of security. In the example that Figure 1 shows, you can see that the local computer has negotiated a Triple DES (3DES) encrypted session with a computer named w2kad.

You can use IP Security Monitor to determine whether Win2K is using a policy and to verify the level of security. But how can you be absolutely sure that data traveling over the network is encrypted? Also, you might use IPSec policies to implement strong authentication methods that restrict communications to certain authorized computers. But how can you prove that these policies are effective? In both cases, the answer is testing.

To prove that data truly is encrypted, install a sniffer, such as Win2K's Network Monitor, to capture packets between the two IPSec-enabled computers. (For information about installing Network Monitor, see the Microsoft article "How to Install Network Monitor in Windows 2000" at http://support.microsoft.com/default.aspx?scid=kb;en-us;q243270.) The sniffer should be able to determine only the packet's destination and source address and whether the protocol is IPSec. When the packet is properly encrypted, the sniffer can't determine whether the packet is TCP or UDP or reveal header information such as port numbers—much less reveal your data.

To verify that your authentication methods are working as expected, try to connect from an unauthorized computer to a protected computer. For example, if you use preshared key authentication, try to connect to a protected computer from a computer at which you've entered the wrong key; the connection should fail. (You can use the Ping command for this attempt.) If you use Kerberos, try connecting from a computer that isn't in your forest or in an external trusted domain; again, the connection attempt should fail. If the attempt succeeds, go back to your policy and determine whether your filter is catching the correct traffic.