Using IPsec to Isolate Servers in Windows 7, Windows Server 2008, and Windows Vista

The primary purpose of IPsec is to protect the content and integrity of network traffic by implementing digital signing and encryption. But when you need to restrict access to certain resources, you probably turn to Access Control Lists (ACLs) or VLANs as the most likely candidates for this purpose. But ACLs—specified in the application layer—can end up posing a great security risk. In reality, you can use IPsec for purpose of isolating specific hosts or domains from the threat of unauthorized (or unmanaged) computers.

In particular, the new IPsec-based Connection Security Rules in Windows Server 2008, Windows Vista and Windows 7—configurable through the Windows Firewall with Advanced Security Console and Group Policy—provides an excellent tool for implementing server isolation. Let's start with a little background about server isolation in general, then dive into the process of configuring it in your environment.

What Is Server Isolation?
By implementing server and domain isolation, you propagate a network policy that requires that specific servers—members of domain—accept authenticated and secured communications only from other domain-member computers. This network policy isolates specific servers from computers that aren't domain members, or computers that are domain members but don't satisfy certain criteria. For example, you can configure a policy that forces a database server to accept connections only from the servers that are members of a specific security group or that have a specific computer certificate installed.

When you implement isolation this way, there's no need to reconfigure the network or implement any third-party software. Everything you need is already present in the OS. Hosts or domains isolated in this way will require no maintenance in case of changes in network design, or if they're moved to another location or another network device. Because isolation is implemented at the OS level, it won't interfere with other levels of protection.

In Windows Server 2003, server isolation was possible by configuring the Access this computer from network Group Policy setting, but this feature's functionality was limited. It was possible only to grant users or computers the right to access a specific host; you couldn't assign additional options such as the authentication method .Also, it was possible to force an authentication protocol through Group Policy (e.g., to accept only NTLMv2), but it wasn't possible to force Kerberos or to request certificates as a means for authentication.

Server 2008, Vista, and Windows 7 provide new functionalities for server isolation through the Windows Firewall with Advanced Security Console. Aside from providing advanced firewall configuration possibilities, this console lets you implement Connection Security Rules. These rules are crucial for the implementation of server isolation. Although IPsec-based isolation was possible to achieve in earlier OSs such as Windows XP and Windows 2003, Server 2008 and Vista integrate IPsec and firewall functionality for the first time.

Requesting or Requiring?
Your first task is to identify the host you want to isolate, then determine the level of isolation to implement. In some cases, server isolation will occur on all hosts in domain, which essentially equates to domain isolation. However, more often, you'll want to isolate only specific (client or server) machines that require an additional layer of security. So, let's focus on implementing isolation on a single host. (Because Connection Security Rules exist on both Vista and Server 2008, and are configured the same way, I won't focus on a specific OS.)

You'll find the Windows Firewall with Advanced Security Console in the Control Panel Administrative Tools applet. After you open the console, right-click the Connection Security Rules node and select New Rule. Doing so starts the New Connection Security Rule Wizard, which offers several choices. Choose the first option, Isolation. The other available options let you make an exemption rule for specific hosts, implement authentication between two specific computers (the Server-to-Server option), force authentication in tunneling mode (useful for site-to-site links), or make a custom rule.

After you select Isolation and click Next, you must choose between several authentication requirements. Essentially, your choice is between requesting and requiring. If you choose a Request option, authentication will be requested (i.e., offered) for inbound or outbound traffic (or both), but it won't be forced. If the other party can't properly authenticate, traffic will still be allowed. If you choose a Require option, the OS will force authentication and will drop the connection if authentication is unsuccessful. Depending on your required level of security, you can choose Require authentication for inbound connections and request authentication for outbound connections, which is acceptable if you want only to force inbound authentication (when other hosts trying to access this isolated host), or you can choose Require authentication for inbound and outbound connections, which maximizes security by forcing authentication on both inbound and outbound traffic.

The first option, Request authentication for inbound and outbound connections, won't force authentication in any way, so it's not true isolation. The second option, Require authentication for inbound connections and request authentication for outbound connections, will keep an acceptable level of security for an isolated host while still allowing the host to communicate with all other hosts (domain and non-domain). For that reason, the second option is a good solution, so select that option and click Next.

Configuring Authentication
Next, you need to configure an authentication method. In this step, you can actually force Kerberos authentication for a user, a computer, or both; require certificates; or implement a custom authentication method. However, you should be aware that these authentication mechanisms are mandatory at the IPsec level (i.e., the network layer). If some application uses another method of authentication (e.g., NTLM), authentication will occur again at the application layer. If you select the default option, authentication will be implemented as configured on the Windows Firewall with Advanced Security Properties dialog box's IPsec Settings tab. To access these settings, right-click the Windows Firewall with Advanced Security on Local Computer node and select Properties. On the IPsec Settings tab, you can select Customize to configure the values that will be treated as defaults during the creation of new connection security rules.

Computer and user. If you select the second Authentication Method option, Computer and User (using Kerberos V5), every connection attempt to an isolated host will require Kerberos authentication. If both the user and computer are domain members, authentication will occur automatically, requiring no user intervention. This authentication method is easy to implement and provides a high level of security, so I recommend it for most scenarios.

Computer. If you select the Computer (using Kerberos V5) option, authentication will be required only from the computer. In other words, if the computer that's initiating a connection to the isolated host is a domain member, the connection will be permitted without requiring any kind of user authentication.

Computer certificate. The Computer certificate option—the most restrictive—lets you specify that only computers that have a certificate issued by a specific Certification Authority (CA) can access the isolated host.

Health certificates. Near the bottom of the Customize IPsec Settings dialog box is an interesting check box called Accept only health certificates, which can add an additional level of security. If IPsec is used in combination with NAP for unhealthy host isolation, the isolated host will accept only computer certificates that are issued by the Health Registration Authority component of the NAP solution. For more information about configuring NAP with IPSec, see the Microsoft article "Step-by-Step Guide: Demonstrate NAP IPsec Enforcement in a Test Lab" (www.microsoft.com/Downloads/details.aspx?FamilyID=298ff956-1e6c-4d97-a3ed-7e7ffc4bed32).

Advanced. At the bottom of the dialog box, you'll see an Advanced option, which lets you select various authentication methods that are performed during the negotiation and security association phases of IPsec. A security association is a combination of a negotiated key, a security protocol, and a security parameters index, which together define the security used to protect the communication between hosts. When you click Customize, a new window presents twin dialog boxes called First Authentication and Second Authentication.

The first authentication method occurs during the Main Mode phase of IPsec negotiations. In this phase, the two computers establish a secure, authenticated channel by going through policy negotiation, a Diffie-Hellman exchange, and authentication. With the second authentication method, you can specify how the user who is logged on to the peer computer authenticates. Available choices are the Kerberos V5 authentication protocol, user certificates, and a computer health certificate. Both first and second authentication are optional, but for a secure environment, you should require at least first authentication.

For the purposes of this article, select the Computer and User (using Kerberos V5) authentication method and click Next.

Which Network Profile?
On the wizard's Profile page, you can determine which network profile this rule applies to. You choices are Domain, Private, and Public, and they're all selected by default. However, you should consider changing these values, particularly if you're frequently changing the location of the isolated host (e.g., laptop computers). The Domain network profile refers to a network that lets you connect to your domain controllers (DCs) and log on to the domain. The Private profile applies to networks that the user marks as Private (e.g., home networks). The Public profile applies to networks that the user marks as Public after he or she connects to them (e.g., networks with a high security risk, such as networks in hotels and public hot spots).

In some cases, you'll want to clear the Private check box. For example, if you're implementing a Connection Security Rule for your laptop, you'll probably want to isolate it in a domain or a public environment but retain access in your home (private) network. Of course, for maximum security, all options should remain selected.

The final step in the wizard is to give the rule a name. I recommend using a descriptive name. After you click Finish, the rule will be automatically activated and will appear in the list of rules.

Implementing Firewall Inbound Rules
If you want to further restrict communication to the isolated host, you can configure additional firewall inbound rules—for example, specifying the ports that are opened for communication and the IP addresses from which communication will be possible. Although you can do this by using just IPsec or pure networking technologies, the new Windows Firewall with Advanced Security Console lets you control it in from single place. Also, you can require the implementation of encryption for traffic security by implementing Encapsulating Security Payload (ESP) through the Windows Firewall with Advanced Security Console.

Right-click the Inbound Rules node, and select New Rule to start the New Inbound Rule Wizard. Select Port, click Next, choose between TCP or UDP, and specify the port that you want to open (for example, you can open only ports for web traffic, 80 and 443). Click Next. On the next screen, select the Allow connection if it is secure option. Doing so essentially connects this rule with the Connection Security Rule you already created. (Connections through the ports you specify here will only occur if they're authenticated through a Connection Security Rule.) To implement traffic encryption, you can also select the Require the connections to be encrypted option. Click Next, and you can specify the computers and users (domain members) to which this rule applies. Finally, you can choose a network profile and name the rule.

For more information about configuring firewall rules, see the Microsoft article "Windows Firewall with Advanced Security and IPsec for Windows Server 2008" (technet.microsoft.com/en-us/library/dd448524.aspx).

Configuring Exemptions
In some situations, you'll want to exempt computers, groups, or ranges of IP addresses assigned to computers from being required to authenticate when initiating a connection to an isolated host—regardless of other Connection Security Rules. For example, you might use exemption to grant access to infrastructure computers (e.g., AD DCs, DHCP or CA servers) that the isolated host must communicate with before authentications can be performed.

A word of warning: Be very careful configuring isolation rules that can affect infrastructure servers. CA, DC, DHCP, DNS, and other infrastructure servers shouldn't have any requirement for IPsec communication for inbound or outbound connectivity. If rules are created, they should be crafted extremely carefully so that unauthenticated computers can authenticate and get access to these services. Member servers and workstations should be configured to neither request nor require authorization to those servers, and the exception rules should be used to configure that.

To configure exemption, start the New Inbound Rule Wizard again by right-clicking Connection Security Rules, then select Authentication Exemption and click Next. On the next screen, you can click Add to add computers, IP ranges, or specific computer types that will be exempted from authentication. When you make your choice, click Next and select the network profile that this rule will apply to. Then, name the rule and click Finish.

Using Group Policy
The most convenient way to enable server isolation on several computers is through Group Policy. Be aware that, to apply server isolation through Group Policy, you need to have Server 2008 domain controllers (DCs), as these options are not available in Windows Server 2003 Group Policy objects. However, if you have Windows Server 2003 DCs, you can still use IPSec policy options. Before creating and linking a Group Policy Object (GPO), you should group hosts with the same isolation requirements into separate organizational units (OUs).

After you’ve created an OU structure and moved servers to their proper OUs, open the Group Policy Management Console from Administrative Tools. Create a new GPO inside the Group Policy Objects container, right-click it, and select Edit to open it. Navigate to the Computer Configuration node, and expand Policies, Windows Settings, Security Settings, Windows Firewall with Advanced Security. In the Group Policy Management Editor's right pane, you'll see the same UI you’d see if you were running this console locally. Click Connection Security Rules, start the New Inbound Rule Wizard, and implement your desired options as I described earlier.

After you finish, you'll have a Connection Security Rule created inside the GPO. If you right-click the rule, select Properties, and go to the Computers tab, you can specify rule endpoints—computers to which this rule will apply. You can specify one or more computers as either endpoint. You can specify a specific IP address, a subnet, a predefined address, or an IP address range. Be aware that the Connection Security Rule will apply to communications between any computer in Endpoint 1 and any computer in Endpoint 2. After you configured all necessary options, you can link the GPO to the OU that contains the hosts that need to be isolated.

Complementary Security
Server isolation provides an extra layer of security and access control that complements other security technologies such as antivirus, anti-spyware, firewall, and intrusion detection system (IDS) solutions. It lets you use Group Policy settings to create, distribute, and centrally manage Connection Security Rules to isolate specific hosts.

This solution also results in a zero-touch deployment experience and an unchanged experience for end-users. No additional end-user training is necessary, and there's no need to install new software or visit each computer during deployment—a great benefit of this technology!