IPSec and Group Policy: A Stronger Defense

Apply the VPN concept to defend mission-critical resources

As you make your network more porous to support connections to your business partners and customers, you must shore up defenses around the crucial resources on your internal network. Sometimes you can implement internal firewalls to separate your network into zones and accomplish this goal. But what if the traffic or computers that you need to protect don't correspond to convenient physical LAN segments? In such cases, you can take a cue from the Internet and apply the VPN concept to your internal network, using IP Security (IPSec) and Group Policy to shield your mission-critical Windows 2000 servers from attackers who manage to penetrate your perimeter defenses.

The IPSec Advantage
You can use IPSec to secure all IP traffic on your network. The protocol provides authentication, integrity checking, and optional encryption at the packet level—and does so in a way that's transparent to your applications. IPSec authentication is stronger than source—IP address filtering, which is subject to spoofing and is difficult to maintain.

IPSec uses Kerberos, preshared keys, or certificates for its initial authentication. You can assign (i.e., activate) only one IPSec policy at a time on a Win2K computer, but that policy can contain multiple IPSec rules so that the computer treats different kinds of traffic in different ways. An IPSec rule specifies a filter list, action, and authentication method. The filter list catches appropriate packets (according to source IP address, destination IP address, and ports), then subjects those packets to a specified action—Permit, Block, or Negotiate security. The Permit action causes the system to process the packet traffic as if you hadn't implemented IPSec. Block causes the system to drop packets. Negotiate security causes the system to secure traffic using the Authentication Header (AH) or Encapsulating Security Payload (ESP) mode, depending on how you've configured the action. If the system receives a packet that isn't secured by AH or ESP, it sends a message to the originating computer, inviting it to retry the exchange by using IPSec. If the originating computer doesn't respond (because it isn't enabled for or doesn't support IPSec), the receiving computer either acquiesces and drops back to unsecure traffic or rejects communications (depending on the action's configuration). AH mode guarantees both computers that the traffic is authentic, meaning that the computer that claims to have transmitted the traffic truly did so. AH mode also uses integrity checking to make sure the packet wasn't modified in transit. AH is sufficient when you don't care about confidentiality but simply want to limit which computers can communicate with a system and make sure that traffic hasn't been modified in transit. ESP, which is a superset of AH, provides encryption in addition to authentication and integrity checking so that only the receiving system can read the data in the packet. (For more basic information about IPSec, see John Howie, "Securing Wireless Networks," January 2002, InstantDoc ID 23374.)

When you receive an IPSec-secured packet, you know that it came from an authorized computer and hasn't been forged or modified in transit. An IPSec-configured computer that drops unauthenticated packets before they reach your applications can foil attackers: Intruders can't attack an application if they can't communicate with it. With some creative thinking, you can find ways to use IPSec and Group Policy to specify which computers in your domain can communicate with one another, thereby adding security and preventing attacks on mission-critical applications such as SAP, Oracle, PeopleSoft, Microsoft Exchange Server, and Microsoft SQL Server.

A Fine Example
Suppose you want to protect an important Win2K system that runs SQL Server. You want to implement security above and beyond what conventional Win2K and SQL Server options can provide. Out of a network of 5000 user workstations, only 100 workstations need to communicate with the SQL Server system, over port 1433. However, these 100 computers are scattered throughout your network, so restricting traffic to the server according to source IP address is impractical (not to mention unsecure). However, you can use IPSec authentication to limit, in two steps, the computers that can communicate with the SQL Server machine through port 1433.

First, you need to create an IPSec policy on the SQL Server machine to require ESP mode for any traffic on the port. (In this example, I suggest that you use ESP mode to encrypt confidential data traveling between the clients and server.) Then, you need to enable IPSec on the 100 authorized client computers.

Configuring the Server
Open the Microsoft Management Console (MMC) Local Security Policy snap-in on the SQL Server system, and select IP Security Policies on Local Machine in the left-hand pane. Right-click a blank area in the right-hand (aka details) pane, then select Create IP Security Policy from the context menu to launch the IP Security Policy Wizard. Click Next, enter a name such as Secure SQL Server, then click Next again. Clear the Activate the default response rule check box, click Next, then click Finish. (The default response rule causes Win2K to acquiesce to any IPSec request from computers that the system contacts. You want to stay in control of IPSec negotiation in this example, so you need to deactivate the rule.)

The Secure SQL Server Properties dialog box opens automatically. On the Rules tab, click Add to launch the Security Rule Wizard. Click Next until you reach the Authentication Method screen. Keep in mind that you're configuring computer-to-computer authentication, not user authentication. When all the computers involved are part of an Active Directory (AD) forest, Kerberos is the easiest authentication method to use because each computer already has a Kerberos-enabled AD computer account. Kerberos isn't as secure as the other options (i.e., certificates and preshared keys), but it's much less work. Therefore, let's start by examining how to use Kerberos authentication. Select the Windows 2000 default (Kerberos V5 protocol) option.

Click Next to advance to the wizard's IP Filter List screen. A filter list contains one or more filters that you configure to catch specific types of traffic and to handle that traffic according to actions that you specify. Click Add to open the IP Filter List dialog box. In the Name box, enter SQL Server Traffic (port 1433), then click Add to launch the IP Filter Wizard. Click Next, then select Any IP Address from the Source address drop-down list. Click Next, then select My IP Address from the Destination address drop-down list. Click Next, select TCP from the Select a protocol type drop-down list, then click Next again. Select the To this port option and enter 1433 in the text box. Click Next, clear the Edit properties check box, then click Finish to return to the IP Filter List dialog box, which will now look like the dialog box that Figure 1 shows. Click Close. On the Security Rule Wizard's IP Filter List screen, select SQL Server Traffic (port 1433), then click Next to advance to the Filter Action screen.

You're now ready to select an action. The prebuilt filter actions won't suffice because none of them make ESP mandatory; therefore, you need to create a custom action. On the Security Rules Wizard's Filter Action screen, click Add to launch the Filter Action Wizard. Click Next. Enter a name such as Require ESP Mode, then click Next. On the Filter Action General Options screen, select the Negotiate security option, then click next. Select Do not communicate with computers that do not support IPSec, then click Next. On the IP Traffic Security screen, select the High (Encapsulated Security Payload) option, click Next, then click Finish to return to the Security Rule Wizard's Filter Action screen. Select Require ESP Mode, click Next, then click Finish. On the Secure SQL Server Properties dialog box, clear the selected SQL Server Traffic (port 1433) check box, then click Close.

You've created the Secure SQL Server policy, but don't assign it yet. Doing so would immediately prevent clients from communicating with the server because you haven't yet configured the clients for IPSec.

Configuring the Clients
To configure the 100 client computers, you need to create a group, add the client computers to the group, create a Group Policy Object (GPO) in AD, and modify the GPO's ACL to give the Read and Apply Group Policy permissions only to members of the new group. After you configure the GPO and enable IPSec for the GPO, and after the client computers apply the GPO, they'll be able to communicate with the SQL Server system. (Other systems will be shut out after you activate the policy on the SQL Server machine.)

Open the MMC Active Directory Users and Computers snap-in, and select the domain root or organizational unit (OU) in which you want to create the group. Select Action, New, Group from the menu bar. Enter Authorized SQL Server Clients in the Group name text box, click Next, then click Finish. Open the new group's Properties dialog box, and go to the Members tab. Add the 100 client computers to the group as members, then click OK.

Now you're ready to create the GPO. Open the domain's Properties dialog box and go to the Group Policy tab. Click New to create a new GPO in the Group Policy Object Links list. Name the GPO Authorized SQL Clients IPSEC.

Typically, when you link a GPO to a domain root, Win2K applies that GPO to all the computers and users in the domain. However, you want the IPSec policy in this new GPO to apply only to the 100 or so computers that are authorized SQL Server clients. You could create a new OU to hold those computers, then link the GPO to the OU—thus limiting application of the GPO. But the SQL Server client computers are already scattered throughout existing OUs. Other GPOs link to these OUs and supply other important policies for the computers. Therefore, you need another way to limit the new GPO to the appropriate computers. The solution is to modify the GPO's ACL.

Open the GPO's Properties dialog box and go to the Security tab. This tab displays the GPO's ACL, which controls who can edit the GPO and provides a way to limit the computers or users to which the GPO applies. Notice that the default ACL grants the Read and Apply Group Policy permissions to the Authenticated Users group. Select that group, then click Remove. Click Add, select the Authorized SQL Server Clients group, click Add again, then click OK. Select the group, then select the Read and Apply Group Policy permissions, as Figure 2 shows. Click OK to close the Authorized SQL Clients IPSEC Properties dialog box. Now, the GPO will apply only to computers that are members of the Authorized SQL Server Clients group even though you've linked this GPO to the domain root.

You now need to configure an IPSec policy for the GPO. On the Group Policy tab of the domain's Properties dialog box, select Authorized SQL Clients IPSEC, then click Edit. Select Computer ConfigurationWindows SettingsSecurity SettingsIP Security Policies on Active Directory. Right-click Client (Respond Only) in the details pane, then select Assign from the context menu. This prebuilt policy causes a Win2K computer to acquiesce when it tries to communicate with another computer that requests IPSec negotiation. Wait at least 2 hours for all the clients to update Group Policy. (By default, Win2K computers reapply Group Policy every 90 minutes, with a random offset of up to 30 minutes.)

You can now activate the Secure SQL Server policy on your SQL Server system. To do so, open the Local Security Policy snap-in on the SQL Server system, right-click the policy (under IPSec Policies on Local Machine), then select Assign from the context menu. Now, only authorized computers can connect to port 1433 on your SQL Server system, and IPSec will encrypt traffic on that port as it traverses the network.

Authentication Alternatives
As I mentioned earlier, for our sample scenario (in which we're limiting the computers within a forest that can communicate with one another) Kerberos authorization is often the simplest—but not the strongest—option. When you use Kerberos, anyone with Administrator access to a computer in the forest need only assign the Client (Respond Only) policy on that computer to attack the SQL Server system through port 1433. In such a scenario, preshared key authentication is a better alternative than Kerberos. To use preshared key authentication, you need to configure both the server and client IPSec policies with a secret key.

Open the Local Security Policy snap-in on the SQL Server system. Right-click the Secure SQL Server policy (under the IP Security Policies on Local Machine object), then select Unassign so that you won't interrupt communications with clients.

Open the policy's Properties dialog box. On the Rules tab, select the SQL Server Traffic (port 1433) rule from the IP Security Rules list, then click Edit to open the Edit Rule Properties dialog box. Go to the Authentication Methods tab and remove the Kerberos entry. Click Add. On the New Authentication Method Properties dialog box, select the Use this string to protect the key exchange (preshared key) option and enter a string of numbers, symbols, and letters at least 20 characters long. Make a note of this string, then click OK three times to close all the dialog boxes.

Next, open the Active Directory Users and Computers snap-in and open the domain's Properties dialog box. Go to the Group Policy tab, select Authorized SQL Clients IPSEC, then click Edit. Select the Computer ConfigurationWindows SettingsSecurity SettingsIP Security Policies on Active Directory object. The Client (Respond Only) policy is assigned. The simplest way to change this assignment is to edit the policy and add a new authentication method. However, I don't recommend this approach because GPOs share IPSec policies. If you use a given IPSec policy, such as Client (Respond Only), in more than one GPO and you change the policy, those changes will take effect in all the GPOs to which you've assigned the policy. Instead, right-click a blank area in the details pane and select Create IP Security Policy from the context menu to launch the IP Security Policy Wizard. Click Next, name the new policy Authorized SQL Clients, then click Next. Select the Activate the default response rule check box, then click Next. Select Use this string to protect the key exchange (preshared key), then enter the same key you entered for the Secure SQL Server policy, as Figure 3 shows. Click Next, then click Finish. Click OK to close the Properties dialog box. Right-click the new policy, then select Assign from the context menu; this action also unassigns the Client (Respond Only) policy.

Wait at least 2 hours for all the clients to reapply Group Policy. After 2 hours, you can reassign the policy on the SQL Server system. At that point, no computer will be able to connect to your SQL Server system unless that computer is a member of the Authorized SQL Server Clients group.

The Next Step
Preshared key authentication also has some weaknesses. Preshared keys are stored in clear text in the registry and are therefore subject to compromise. Also, you've protected only port 1433. What about other ports that an attacker could target, such as those associated with the Server service or Windows Terminal Services? For this sample scenario, the strongest authentication option—albeit the most complicated—is certificates. In a follow-up article, I'll show you how to set up a Certificate Authority (CA) and configure IPSec to use it to lock down access to our sample SQL Server system. I'll also shed more light on sequencing changes to IPSec to make sure you don't temporarily interrupt communications while Win2K applies Group Policy throughout your domain. Until then, examine your network and consider other ways in which you can use IPSec to erect defenses behind your firewall. After all, to secure an office building, you don't just lock the front door, you also lock the offices that contain crucial equipment to protect against malicious insiders as well as outsiders who make it past your front door. Likewise, don't give up your whole network just because someone makes it through your firewall—use IPSec to limit communication with your mission-critical servers.