IPSec Tunneling with ISA Server

Some research companies, notably IDC, predict that nearly 100 percent of Internet traffic will be encrypted by 2005. Although a portion of this traffic will consist of credit card transactions, pretty good privacy (PGP)–encrypted email, and encrypted file transfers, most will be protected because of the increasing use of VPNs—between company sites, business partners, or the office and employees' homes. Many companies that use Microsoft Internet Security and Acceleration (ISA) Server 2000 as a firewall and proxy server ask me whether they can also use the product to establish an Internet VPN. The product's robust and useful wizards can indeed help you quickly establish a client-to-gateway or gateway-to-gateway VPN. Still, setting up an ISA Server VPN involves many steps as well as Certificate Services, DHCP, DNS, and RRAS, so the process is more complex than just setting up a firewall.

VPN Tunneling
If you've never set up a VPN, the way in which they work can seem a bit awkward. First, you must use hardware or software to establish endpoints—one or more VPN clients and a VPN server. You typically can establish a VPN's physical and data-link layers (i.e., Layers 1 and 2 of the Open System Interconnection—OSI—model) over a dial-up line or a high-speed dedicated digital line. The endpoints don't need to use the same vendor's solution, but they must use the same tunneling protocol. Most VPN implementations use Layer Two Tunneling Protocol over IP Security (L2TP/IPSec), as the sidebar "ISA Server VPN Protocols," page 2, explains, and can partner with one another, but interoperability conflicts still abound, especially for ISA Server VPNs. (Most vendors' L2TP implementations vary at least slightly, like different dialects within a language.) The key to interoperability is that both endpoint solutions must support the same IPSec protocol and configuration options. (See the Web-exclusive sidebar "IPSec Protocols and Modes," http://
www.winnetmag.com/windowssecurity, InstantDoc ID 40596, for a discussion of IPSec protocols.)

You can configure a client-to-gateway VPN between a client and a server, or you can configure a gateway-to-gateway (aka site-to-site) VPN between two or more VPN network endpoints (servers or clients). ISA Server's two most common Internet VPN scenarios are between a Windows PC client and an ISA Server (client-to-gateway) and between two ISA Servers (gateway-to-gateway).

Clients at either end of a gateway-to-gateway VPN maintain their own identities and traffic domains, but remote clients on a client-to-gateway VPN become virtual hosts on the VPN server's network. The process of establishing a VPN connection often assigns these clients new IP addresses, subjecting the clients to the same traffic as physical hosts on the network. Therefore, remote clients often lose their local networking services and might not be able to connect to their local servers, printers, or Internet services. This type of configuration is known as a tunnel-mode VPN; remote clients in a tunnel-mode VPN have access only to network servers and printers and often access the Internet through the VPN. You can configure your DHCP server or RRAS to assign valid network IP addresses to remote clients. With the first method, RRAS gets a range of addresses from the DHCP to pass to the VPN clients; with the second method, RRAS assigns the addresses from its own DHCP server pool. If you need to pass advanced DHCP scope options to the remote clients, you must set up a DHCP relay agent on the RRAS computer. I suggest that you use a dedicated subnet for your VPN clients so that you can easily distinguish between VPN and LAN clients.

Alternatively, split-mode VPNs give remote clients simultaneous local and network access. Split-mode clients, however, can become unsecured gateways and can introduce rogue traffic into the VPN. Some VPN clients, including the Windows Network Connection VPN client, support both tunnel and split modes. The mode you choose depends on your objectives and your VPN solution.

The clients on either side of a VPN transmit unencrypted information; the VPN endpoints—typically routers, firewalls, or ISA Server systems, as Figure 1 shows—perform all encryption. The application in which the user created the data is usually unaware of the VPN. The VPN hardware or software decrypts the encapsulated data at the other end of the VPN, so if someone were to capture the encapsulated traffic, the most that person could read would be the IP header, and even that header usually isn't the original. IPSec protects TCP and UDP headers, source and destination addresses, and packet payload content.

To support L2TP/IPSec, you might need to open certain ports on any involved firewalls. You'll probably need to open UDP port 1701 for L2TP traffic, UDP port 500 for Internet Key Exchange (IKE) traffic, and UDP port 4500 for Network Address Translation Transversal (NAT-T) traffic if you use NAT-T. (See the Web-exclusive sidebar "NAT Transversal," http://www.winnetmag.com/windowssecurity, InstantDoc ID 40597, for more information about NAT-T.) Configuring IPSec connections on ISA Server creates the appropriate inbound and outbound packet filters for these ports, although you might need to open additional ports for additional client access (e.g., DHCP, DNS, NetBIOS). You'll probably also need to permit IP Type 50 Encapsulating Security Payload (ESP) packets and IP Type 51 Authentication Header (AH) packets. Also be prepared to use preinstalled machine certificates or to install certificates. (See the Web-exclusive sidebar "Certificate Authentication," http://www.winnetmag.com/windowssecurity, InstantDoc ID 40598, for information about certificates.) Now that you have a bit of background about how client-to-gateway and gateway-to-gateway ISA Server VPNs work, let's look at the steps involved in setting up these popular VPNs.

Client-to-Gateway VPNs
Employees working from home, laptop users on the road, and Internet-based Terminal Services users are common users of client-to-gateway VPNs between a remote Windows client system and an ISA Server system. If the ISA server is a member server, remote clients can use established domain user accounts; if the ISA server is a standalone server, you must create user accounts on the ISA server for each VPN user.

After you've created any necessary user accounts, you must configure the ISA server to accept VPN client connections, then configure the clients to connect to the ISA server. On the ISA server, perform the following steps:

A VPN connection to an ISA server from a remote client is actually two connections. The first connection establishes the physical link (usually to the client's ISP); the second connection establishes a link to the ISA server. To configure a Windows XP or Windows 2000 VPN client, follow these steps on the remote client system:

For clients running earlier Windows versions, you must install and configure the Microsoft L2TP/IPSec VPN Client. Installing this client also installs a new remote access device, called RASL2TPM, on Windows NT systems and installs the Microsoft L2TP/IPSec VPN Adapter on Windows Me and Windows 98 systems. NT systems must run Service Pack 6a (SP6a), Microsoft Internet Explorer (IE) 5.01 or later, PPTP, and RAS. Win98 systems must run IE 5.01 or later and the DUN 1.4 upgrade. Windows Me systems must run IE 5.5 or later and the VPN networking component, which is listed as VPN network adapter under Network properties. (IE must be installed on these systems but doesn't need to be active or configured as the default browser.)

Gateway-to-Gateway VPNs
To establish a gateway-to-gateway VPN between two ISA Server systems, you must configure both systems, although a configuration file that the first (local) system creates during its setup simplifies the setup of the second (remote) system. Perform the following steps:

Make the configuration file and the local ISA server's X.509 certificate available to the remote ISA Server computer, or copy the file and certificate to the remote system. Then, on the remote ISA server, complete the following steps:

Steps to Security
Opening your company's private network to Internet VPNs increases the risk of a security exploit. ISA Server versions earlier than SP1 always trust VPN clients and therefore filter nothing between the remote client and the network. Therefore, any security threat on the VPN client has a clear channel around ISA Server's firewall filtering and onto your LAN. ISA Server SP1 applies packet filtering to most VPN connections, but RRAS dial-up VPN interfaces remain unfiltered. Before you implement an ISA Server VPN, I recommend that you study the related security concerns and read the Microsoft article "How to Configure IPSec Tunneling in Windows 2000" (http://support.microsoft.com/?kbid=252735) for more details about packet filtering between VPN endpoints.

ISA Server is a popular choice for Internet VPNs; the product uses L2TP/IPSec, which is becoming the world's VPN authentication and encryption standard, to connect two networks or let remote PCs connect to a corporate LAN. ISA Server's wizards walk you through a lot of the work of setting up a VPN, but you still need to know quite a bit about ISA Server, DHCP, DNS, RRAS, and VPNs, and to maintain network security, you must take precautions to filter VPN traffic beyond the defaults. For more information about these topics, see "Related Articles."