How to Implement SDI Using IPSec in a Mixed Environment

Many enterprises proactively defend against malware outbreaks by limiting access to their key Windows servers and networks. One popular defense is Server and Domain Isolation (SDI) using IPsec. SDI is easy to implement in Windows environments. However, some enterprises maintain a heterogeneous environment of Windows, UNIX, and Linux systems. Fortunately, you can extend SDI to UNIX and Linux systems.

One reason why SDI is easy to implement in Windows OSs is that you can use Kerberos as an authentication mechanism. However, not all UNIX and Linux clients support Kerberos. Most UNIX and Linux clients do support preshared keys (e.g., passphrases, sequences of random characters) and X.509v3 certificates as authentication mechanisms. Using preshared keys is the simplest authentication method to implement, but it's generally considered the least secure. The use of certificates is more secure but requires a public key infrastructure (PKI).

To keep things simple, I'll show you how to use preshared keys to configure three popular UNIX and Linux OSs to use IPsec so that SDI is supported for both outgoing and incoming connections. I'll only cover how to configure your Windows systems to the extent necessary for interoperability with the UNIX and Linux systems. For more information about configuring IPsec to support SDI on Windows systems, see the references in the Learning Path at the top of the page.

Before I proceed, I must caution you that incorrectly configuring and deploying IPsec in your Windows environment, especially when using Group Policy, can break all communications between systems and between systems and domain controllers (DCs). I recommend you test everything in a lab environment before deploying IPsec to production systems and networks.

Preparing Your Windows Environment

Before you start to configure your UNIX and Linux systems, I recommend that you configure SDI in your Windows environment first. By default, Windows uses Kerberos as the authentication method for IPsec. If your UNIX and Linux systems don't support Kerberos as an authentication method, you don't need to stop using Kerberos with IPsec in your Windows environment. IPsec will negotiate a mutually agreeable authentication method. All that's required is that the Windows, UNIX, and Linux systems support the same authentication method.

Most SDI implementations in Windows environments divide systems into secure servers and clients:

Secure servers. Secure servers are configured to always use IPsec in system-to-system communications, typically with exceptions for communications with DCs and default gateways. If a secure server receives an inbound connection that isn't secured using IPsec and the connection isn't from a DC, a default gateway, or another exempted system, the server will reply that IPsec is required. You can view these replies using the Microsoft Management Console (MMC) IPsec Security Monitor snap-in or Network Monitor.

Note that DCs are exceptions because Windows uses Kerberos by default, and DCs authenticate users and computers. If you can't communicate to a DC without IPsec, which requires authentication, you can't be authenticated. In addition, DCs typically offer services such as DHCP and DNS. DHCP is used to obtain an IP address, so IPsec can't be used before the machine has fully started up and authenticated itself against a DC. DNS is used for DC discovery for authentication. IPsec can't be established before authentication.

Clients.Clients are typically configured to attempt all communications without IPsec. When they attempt to communicate with a secure server, the server will reply that IPsec is required. In response, the client will begin to authenticate using IPsec. Once authenticated, communications will continue.

The simplest means to configure and manage IPsec policies in your clients and secure servers is to use Group Policy Objects (GPOs). From this point on, I assume that you already have GPOs with IPsec policies for your clients and secure servers.

In the Group Policy Object Editor, you need to perform the following steps for each GPO:

Changes to policies will be distributed to your Windows systems at the next policy refresh interval or when the Windows systems start up, whichever comes first. For clients with more than one IPsec rule and for secure servers, you'll need to add an authentication method for each IPsec rule.

Preparing UNIX and Linux Systems for IPsec

The steps you need to take to prepare your UNIX and Linux systems for IPsec will vary, depending on your UNIX or Linux distribution. Sun's Solaris 10 comes with full support for IPsec, so you don't need to prepare that system for configuration. Most Linux distributions don't provide a full implementation of IPsec out of the box, so you'll likely need to install additional support to the kernel. Check your installation media for IPsec tools (e.g., ipsec-tools.rpm). If you don't have any tools, you can download and build IPsec tools for Linux Kernel 2.6 from the IPsec-Tools web page.

On BSD variants of UNIX, you need to make sure support for IPsec is included in the kernel. There are a couple of implementations available. I'll describe the most common implementation here, which is available on FreeBSD.

To check whether support for IPsec is included in the kernel, you need to check the options section of the kernel configuration file in /sys//conf directory, where is your machine architecture (e.g., i386). You can use the command

uname –p 

to determine your machine architecture. Note that FreeBSD commands are case sensitive.

To find out the name of the kernel configuration file, type the command

uname –I 

The current kernel configuration file will have a value that matches the value reported by uname in its ident entry. After you have the name of the kernel configuration file, type the command

grep IPSEC  

where is that filename. The output should have the following two entries:

options   IPSECoptions   IPSEC_ESP

If one or both of these entries is missing, you need to add them to the options section. After adding those entries, you must build a new kernel by running the commands

config cd ../conf/make cleandepend; make dependmake

After you've built your kernel, type

make install

then reboot the system. You also need to install an Internet Key Exchange (IKE) daemon. FreeBSD has one named racoon in the /usr/ports/security/ipsec-tools directory. To download racoon, type

make

in the ports directory. In the options menu that appears, click OK to accept the defaults and to download the port to your system.

Planning for the Configuration

After the necessary software is in place on your UNIX and Linux systems, you need to plan for the configuration those systems. For example, you need to:

The process for configuring each UNIX or Linux variant in your environment will likely differ. However, some elements of the process will be common across all of them. In the configuration examples that follow, I assume you're using Encapsulating Security Payload (ESP) NULL, in which the contents of packets are authenticated but not encrypted. ESP NULL is commonly used in SDI solutions to authenticate end systems communicating with each other. If you want to use ESP and/or Authentication Header (AH), the examples will help you get started.

Configuring Solaris 10

Configuring Solaris 10 to use IPsec involves configuring the IPsec rules, creating the IKE configuration file, adding the preshared keys to the ike.preshared file, and starting the IKE daemon.

Configure the IPsec rules. The first step is to configure the IPsec rules, much like you would do in a Windows system. This is typically done by adding rules to the ipsecinit.conf file, which you will find in /etc/inet. The rules in this file are read at each boot. The format of each rule is

{ pattern } action { properties }

where { pattern } is the set of conditions that causes the rule to fire, action is the action that the rule should perform, and { properties } is the set of arguments or properties for the action. The action is usually ipsec (requires IPsec to be used) or bypass (doesn't require IPsec to be used). There are other actions such as drop, permit, and pass. For more details on these and other actions, consult the manual (aka man) page ipsecconf(1M).

For example, if all your Windows secured servers are on the subnet 172.21.23.0/24 and the servers require ESP NULL with Secure Hash Algorithm-1 (SHA-1), the rule would look like

{ raddr 172.21.23.0/24 dir both } ipsec { encr_algs null encr_auth_algs sha1 }

You can have multiple conditions in the { pattern } element of the rule to narrow the match, including source and destination addresses, source and destination ports, and protocols. If you have multiple conditions in the { pattern } they are logically AND'ed together and all must be true for the condition to fire. You might need multiple rules if you have multiple subnets with secured servers, want to secure communications to individual services listening on known ports, or want to create exemptions such as for Internet Control Message Protocol (ICMP). Rules are processed in the order in which they appear in the file, with the exception of bypass rules, which are always processed first, so you need to take care in ordering the rules. I recommend that you place narrow or specific rules before broad rules in logical groupings (e.g., by subnet). For more information about adding rules to the ipsecinit.conf file, see the man page ipsecconf(1M). In addition, you can find a useful sample file named ipsecinit.sample in the /etc/inet directory.

After you add your IPsec rules to the configuration file, you can run the following command to load them:

ipsecconf –a /etc/inet/ipsecinit.conf 

Note that Solaris commands are case sensitive. You can view loaded rules at any time by simply typing

ipsecconf

at the command line.

Create the IKE configuration file. The next step in configuring Solaris is to create the IKE configuration file, which by default is named config and stored in the /etc/inet/ike directory. When this file is present, the IKE daemon starts at boot time. There is a sample IKE configuration file named config.sample in the /etc/inet/ike directory, but I don't recommend using it because it uses certificates. Instead, edit the config file so that the first four lines look identical to the four lines in Listing 1. (To obtain a text file containing these lines, click the Download the Code Here button at the top of the page. The Listing_01.txt file will be in the 101264.zip file.)

These are the defaults used when the Solaris system negotiates with other systems to use IPsec. These defaults will work with typical Windows IPsec policies. If needed, you can override the default configuration. More details can be found in Chapter 23, "Configuring IKE (Tasks)" of the System Administration Guide: IP Services for Solaris.

Besides the four lines in Listing 1, you need to add to the IKE configuration file a line for each remote system that Solaris will communicate with using IPsec. The format of each line is

{ label "" local_addr  remote_addr  }

where is a unique text description, is the IP address of your Solaris system, and is the IP address of the remote system your Solaris system will communicate with.

Add the preshared keys to the ike.preshared file. At this point, you need to add the preshared keys to the ike.preshared file, which is in the /etc/inet/secret directory. You need to add an entry for each remote system that your Solaris system will communicate with. Each entry needs to take the format

{ localidtype IP localid  remoteidtype IP remoteid  key  } 

where is the preshared key. The preshared key must be entered as a hexadecimal value. This contrasts with Windows, where the preshared key is entered as ASCII text. For example, the preshared key 1234567890ABCD0987654321 on Windows is 313233343536373839304142434430393837363534333231 on Solaris.

Start the IKE daemon. The last step is to start the IKE daemon. This is done by running the command

/usr/lib/inet/in.iked

After the daemon is running, your Solaris system can communicate with each remote system configured in the /etc/inet/ike/config and /etc/inet/secret/ike.preshared files, as long as a rule exists in the ipsecinit.conf file. The daemon will start at boot time due to the presence of the /etc/inet/ike/config file you just created.

Configuring FreeBSD

The process of configuring FreeBSD for IPsec is somewhat similar to Solaris in that there is a heavy reliance on configuration files and policies to control IPsec, but the FreeBSD configuration is more flexible. Configuring FreeBSD involves creating the racoon.conf file, creating the racoon.presharedkey file, creating and loading the security policies, starting the racoon daemon, and loading the security policies.

Create the racoon.conf file. The first step in configuring IPsec on FreeBSD is to create the racoon.conf file in the /usr/local/etc/racoon directory. Edit the file so that it contains the contents of Listing 2. (A text file of Listing 2 is in 101264.zip.) This listing is generic enough to work with most Windows IPsec policies in which ESP NULL is used.

Create the racoon.presharedkey file. The next step is to create the racoon.presharedkey file in the /usr/local/etc/racoon directory. As callout A in Listing 2 shows, the /usr/local/etc/racoon/racoon.presharedkey pathname is specified in the racoon.conf file.

In racoon.presharedkey, you need to add a line for each remote system your FreeBSD system will communicate with. Each line needs to contain the remote system's IP address and the preshared key that will be used for the IPsec communications. The lines need to follow the format

where is the IP address and is the preshared key. The preshared key can be alphanumeric or hexadecimal. Alphanumeric preshared keys can contain spaces and don't need to be surrounded by double quotes. Hexadecimal keys must be prefixed with 0x. In the following example, the same preshared key (one in ASCII and one in hexadecimal) is used when communicating with two different remote systems:

172.21.23.128   1234567890ABCD0987654321172.21.23.129   0x313233343536373839304142434430393837363534333231

The owner (which should be the root user) should have read-only rights to racoon.conf file. No one else should have rights to access this file.

Create and load the security policies. Next, you need to create the security policies that IPsec will use to filter incoming and outgoing packets and to determine when to establish IPsec communications. Policies specify rules for incoming or outgoing packets, but not both, and can identify a host or subnet of hosts. The policies are stored in the kernel and are loaded using the setkey(8) command, either interactively or from a file.

Because policies aren't persisted across system reboots, the simplest means to add policies to the kernel is to create a file that can be read by setkey in a startup script. To do so, you need to create the /usr/local/etc/setkey/setkey.conf file and add the following policies for each host or subnet range you want to use IPsec with:

spdadd 0.0.0.0/0  any –P out ipsec esp/transport//require ;spdadd  0.0.0.0/0 any –P in ipsec esp/transport//require ;

These policies assume you're using ESP NULL only. To load the policy, use the command

setkey –f /usr/local/etc/setkey/setkey.conf

Start the racoon daemon. After the policies are loaded, you can start the racoon daemon and test IPsec connectivity. Start the racoon daemon by typing

/usr/local/sbin/racoon

at the command line. You can now test your policies to ensure they work. The racoon daemon will start at boot time if you add the following line to the /etc/rc.conf file:

racoon_enable="YES"

Load the security policies. The last step in configuring FreeBSD for IPsec is to load the security policies into the kernel at boot time. This is most easily done by creating the /usr/local/etc/rc.d/setkey file, which should contain the contents in Listing 3. (A text file of Listing 3 is in 101264.zip.)

After you create this file, add the following line to the /etc/rc.conf file:

setkey_enable="YES"

The next time the FreeBSD system boots, the security policies will be loaded.

Configuring Linux Systems

Configuring Linux is remarkably similar to configuring FreeBSD, with the main difference being the locations of the configuration files. In Linux, both the racoon.conf and setkey.conf files are in the /etc/racoon directory. This directory also contains sample racoon.conf and setkey.conf files, which were created when you installed the IPsec tools. Edit these sample files using the directions and listings I provided in the section on configuring FreeBSD. However, there are two changes you need to make to the racoon.conf file in Listing 2:

To configure the racoon daemon to start at boot time so that the security policies will automatically load, you need to create links to /etc/rc.d/racoon in /etc/rc.d/rc3.d and /etc/rc.d/rc5.d for startup and shutdown. For example, the command to create the links might look like

cd /etc/rc.d/rc3.dln –s ../racoon S15racoonln –s ../racoon K15racooncd ../rc5.dln –s ../racoon S15racoonln –s ../racoon K15racoon

Note that Linux commands are case sensitive. The racoon startup and shutdown scripts manage the security policies for you, so you don't need to create separate setkey startup and shutdown commands. You can affect the order that racoon starts and stops by changing the number after S and K.

Troubleshooting

IPsec is not simple to configure. You might find that you have to troubleshoot configurations, preshared keys, and policies before you can get two systems to communicate when IPsec is required. This will likely be the case in complex configurations in which you're using AH and ESP in addition to ESP NULL.

To help you troubleshoot, you can use Network Monitor 3.2, which runs on Windows Server 2008, Windows Server 2003, Windows Vista, and Windows XP. Network Monitor lets you see IKE packets being exchanged between Windows systems and UNIX and Linux systems. You can also see the proposals being made by the end systems, which lets you see if the correct proposals are being offered based on your choices in GPOs. You can download this free utility from the Microsoft Download Center.

On Solaris, you can start in.iked in debug mode by specifying –d on the command line. In debug mode, the daemon won't disassociate from the terminal and will log output to standard output. You'll see a wealth of information that lets you troubleshoot why IPsec communications are failing.

On FreeBSD and Linux systems, you can start racoon in the foreground by specifying

racoon –F 

on the command line. Adding the –d and –v flags will produce a lot of debug information that you can use. You can add –l and the name of a log file to have racoon log to a file. The –v, -d and –l flags can be used when racoon runs in the foreground or background. The returned information includes a proposal-by-proposal comparison provided by the remote system with acceptable proposals defined on the local system, which is very useful in ensuring that there are matching proposals. You can check which policies are loaded into the kernel by running the command

setkey –DP

You can also check for established Security Associations (i.e., successful IPsec associations) by running

setkey –D

from the command line.

Use Preshared Keys to Establish IPsec Communications

As you have seen, you can use IPsec with preshared keys to establish IPsec communications between Windows and common UNIX and Linux variants. Although using preshared keys isn't considered as secure as using certificates or Kerberos, preshared keys are supported by a wide range of systems. In addition, preshared keys are easier to configure than certificates or Kerberos. By using IPsec with preshared keys, you can extend SDI to your UNIX and Linux systems.