IPSec Protocols and Modes

Two primary types of IP Security (IPSec) protocols exist: IP Type 50 Encapsulating Security Payload (ESP) and IP Type 51 Authentication Header (AH). ESP provides authentication and encryption; AH provides authentication but not encryption. Microsoft’s IPSec implementation uses Data Encryption Standard (DES) or Triple DES (3DES) for encryption.

ESP encrypts the TCP or UDP header along with the payload data. Whether IPSec modifies the original IP header depends on the IPSec mode. IPSec supports transport or tunnel mode, both of which can use either ESP or AH packets. Transport mode secures packets between two endpoints, typically in a client-to-gateway scenario, and leaves the original IP header unchanged. Tunnel mode encapsulates the IP header and payload into a new IPSec packet for transfer between two endpoints, typically two IPSec gateway devices. In either mode, both IPSec endpoint devices use Internet Key Exchange (IKE) to negotiate authentication and use encryption if necessary to support ESP. Layer Two Tunneling Protocol (L2TP) and IPSec work together by placing the content payload—an L2TP packet—within an IPSec packet. IPSec contributes the encryption and machine authentication mechanisms, and L2TP provides user authentication and multiprotocol support.

IPSec, which is built into Windows Server 2003, Windows XP, and Windows 2000, integrates with Active Directory (AD). Win2K uses group and local policies to implement IPSec policies. To use L2TP over IPSec (L2TP/IPSec), you must make sure all connection partners' endpoints agree on the protocol (i.e., AH or ESP), the mode (i.e., transport or tunnel), and the authentication method (i.e., Kerberos, preshared secrets, or certificates). The scenarios that I describe in "IPSec Tunneling with ISA Server," http://www.winnetmag.com/windowssecurity, InstantDoc ID 40578, use ESP to encrypt the VPN and use transport mode for client-to-gateway VPNs or tunnel mode for gateway-to-gateway VPNs.