Using IPsec for Network Protection

Preshared keys. Included only for RFC conformance, it's a good idea to use preshared keys only when testing your IPsec policies. Every peer that participates in the same security policy will need the same preshared key. Shared secrets don't remain secret for very long! Furthermore, they're stored in the registry and clearly visible to anyone with administrative privileges on the computer.

Digital certificates. As long as each peer possesses an IPsec or computer certificate signed by an authority the other peer trusts, the peers will authenticate to each other. Note where the trust lies: in the signer of the certificate. The actual name on the certificate is unimportant in this case. Digital certificates are much preferred over preshared keys because each peer can have its own certificate, and a multilevel certificate hierarchy can help create more granular IPsec policies. For example, super-secure Machine A might accept only certificates signed by high-value Authority X, while sort-of-secure Machine B might accept certificates signed either by high-value Authority X or medium-value Authority Y.

Kerberos version 5 protocol. If both peers are in the same Active Directory forest, IPsec on computers running the Windows Server 2003 operating system can also use the Kerberos protocol for the initial computer-to-computer authentication. Kerberos is appropriate if you don't have a public key infrastructure (PKI) and don't need to establish IPsec security associations between computers outside a single forest.

Transport mode. This is the more common of the two and is often what people have in mind when thinking of an "IPsec tunnel." In transport mode, two peers authenticate each other (phase one) and establish the traffic signing and encryption parameters (phase two). Any traffic between the computers that matches the characteristics specified in the filter list will be signed and/or encrypted according to the details of the linked filter action. Transport mode ensures that communications between two computers remains tamper-free and private. Transport mode doesn't create new packets; rather, it secures existing packets. Interestingly enough, in L2TP+IPsec VPNs, it's transport mode, not tunnel mode, that secures the L2TP traffic between a client and a VPN server.

Tunnel mode. Tunnel mode is intended for secure site-to-site communications over an untrusted network. Each site has an IPsec gateway configured to route traffic to the other site. When a computer in one site needs to communicate with a computer in the other site, the traffic passes through the IPsec gateways (and possibly through intervening routers in each site before reaching the local gateway). At the gateway the outbound traffic is encapsulated inside another complete packet and secured according to the details of the filter action in the rule. Of course, the gateways have already performed their phase one authentication and established their phase two signing/encryption security association. In IPsec for Windows Server 2003, tunnel mode is supported only for site-to-site VPNs on Routing and Remote Access Service (RRAS) gateways and not for any kind of client-to-client or client-to-server communications.

AH (authentication header). AH security associations are useful when the requirement is only for integrity but no confidentiality. AH computes a SHA1 or MD5 digital signature across the entire packet (including the IP header which contains the source and destination addresses) and adds this signature to the packet. The receiver computes its own version of the signature and compares that with the signature stored in the header; if they match, the packet hasn't been modified.

ESP (encapsulated security payload). Use ESP security associations when you have a need for confidentiality. ESP will negotiate a DES or 3DES (triple DES) session key that's exchanged between the peers and used for encrypting the traffic between them. You can also specify a SHA1 or MD5 digital signature in ESP. Note that both the ESP encryption and signature computations include the payload and the TCP/UDP header portions of each packet but not the IP header; compare to AH whose digital signature covers the entire packet.

AH integrity violation. AH computes its digital signature of the packet before it leaves the sending peer. If that packet then passes through a NAT (either local or remote), its IP header gets modified. The receiving peer, when it computes its own version of the packet's signature, will generate a different result because the NAT modified the source address. Therefore the receiving peer will drop the packet.

IPsec helper. Many small/home office NAT gateways include a feature called "IPsec helper" (or "IPsec passthrough"). Originally designed for tunnel mode, the feature also acts on transport mode. If multiple computers behind the gateway create IPsec security associations to destinations outside the gateway, the gateway will forward all incoming IPsec traffic to the first computer that created its security association. The helper function simply remembers which computer on the inside initiated an IPsec conversation and forwards all incoming traffic there, without modification.

IKE fragmentation. It's common that the payload of a digital certificate will exceed the size of an IP frame. Whenever an application generates a data packet larger than an IP frame, IP fragments the packet so that each fragment fits in a single frame. While this is fine inside a local network, network border devices (including many NATs) drop fragments because carefully crafted malicious fragments are a popular way to circumvent some firewalls. NATs that drop fragments will prevent IKE from working properly.Turns out, though, that by defining a mechanism to encapsulate ESP (but not AH) inside UDP, it's possible to forward IPsec traffic through a NAT without it getting rejected. Each side sends some discovery packets to the other to determine if there is a local NAT present and if both sides are capable of performing NAT traversal (NAT-T). If one or both sides are behind a NAT and both can perform NAT-T, IPsec will first shift the IKE exchange to UDP port 4500 (to avoid IPsec helper conflicts) and then encapsulate the entire IPsec security association in the same UDP conversation. To the NAT the traffic looks like ordinary UDP traffic, so the NAT can handle it just fine. Each side also exchanges its NAT details with the other so that they can properly reassemble the IP headers of received traffic. Remember that NATs change the headers, so receivers will need to "remanufacture" the original header before decrypting traffic and checking signatures. The NAT-T setup process provides each side with the necessary information to do this.