ISA Server VPN Protocols

Microsoft Internet Security and Acceleration (ISA) Server 2000 supports PPTP and Layer Two Tunneling Protocol (L2TP). PPTP is the less secure of the two protocols and is included primarily for pre–Windows 2000 clients, such as Windows NT and Windows 98, and for networks that use Network Address Translation (NAT).

PPTP authentication and encryption rely on a user ID and password to prevent impersonation. If an intruder can guess the username and password, he or she can read the user's email from a remote machine.

L2TP, which is Microsoft's favorite authentication and encryption protocol, is an extension of PPTP over IP and uses UDP port 1701. L2TP lets you separate the physical and data-link connecting point from tunneling hardware and software. TCP/IP, IPX, and AppleTalk can be sent over L2TP. IP Security (IPSec) works with L2TP to provide end-to-end authentication and privacy. L2TP over IPSec (L2TP/IPSec) is installed by default on Windows Server 2003, Windows XP, and Win2K Server. Microsoft has also released an updated L2TP/IPSec VPN client (http://download.microsoft.com/download/win98/install/1.0/w9xnt4me/en-us/msl2tp.exe) that supports NT, Windows Me, and Win98.

To break into an L2TP/IPSec VPN connection, an intruder needs the user ID and password, plus physical access to the user's PC because L2TP/IPSec also requires a trusted machine certificate. Because L2TP/IPSec is more secure and is now supported on most Windows clients, I use it as the tunneling protocol.

IPSec, which Internet Engineering Task Force (IETF) Request for Comments (RFC) 2662 defines, uses Internet Key Exchange (IKE) to establish standards for secure communications. RFC 2409 defines IKE, which uses UDP port 500, IPSec, IP Type 50 Encapsulating Security Payload (ESP), and IP Type 51 Authentication Header (AH). In Microsoft's IPSec implementation, IKE uses one of three authentication methods to establish trust between IPSec endpoints:

IKE uses one of the above authentication methods and one of two IKE negotiation modes—main mode or quick mode—in two phases to establish security associations (SAs) between IPSec endpoints. IKE main mode is slightly slower than quick mode but is resistant to network sniffing attacks because it uses Data Encryption Standard (DES) or Triple DES (3DES) encryption. IKE quick mode (aka aggressive mode) is often used in limited-bandwidth scenarios but is more easily compromised because it doesn't encrypt initial key information. Phase 1 establishes a secure, authenticated communication channel; you can use either IKE mode, but I suggest you use main mode whenever possible. After Phase 1 establishes the initial secure channel, Phase 2 uses quick mode to negotiate key types, sizes, and protocols. IPSec ensures that the SAs are updated on a regular basis as defined by the IPSec policy. By default, Microsoft's implementation of IPSec updates main mode SAs after 8 hours but renegotiates quick mode SAs every hour.

These authentication methods authenticate IPSec endpoints only to one another. Any encryption that takes place during the IKE exchange phases is intended to protect channel authentication, not to encrypt data. After endpoints authenticate one another, they generate bulk encryption keys for data authentication and encryption.

Windows IPSec clients use Kerberos 5.0 locally to automatically and transparently negotiate IPSec communications. A Kerberos 5.0 Key Distribution Center (KDC) serves each domain or a series of trusted domains within a forest. Kerberos is turned on by default within native Win2K domains, but most VPNs are established between domains or across the Internet, so you often must use preshared secrets or X.509 certificates for IPSec authentication. Preshared secrets are plaintext passphrases that you type at each IPSec endpoint or distribute using IPSec policy and are most often used for quick testing purposes and for interoperability with third-party clients that don't support Kerberos X.509 certificates. (In a Win2K domain, anyone with a valid domain user account can read these passphrases.) See the Microsoft articles "How to Configure a L2TP/IPSec Connection Using Pre-shared Key Authentication" (http://support.microsoft.com/?kbid=240262) and "HOW TO: Configure a Preshared Key for Use with Layer Two Tunneling Protocol Connections in Windows XP" (http://support.microsoft.com/?kbid=281555) for more information about using preshared secrets.