Terminal Services, Part 4

Randy Smith completes his look at features in the Microsoft Management Console (MMC) Terminal Services snap-in. Learn how to use IP Security (IPSec) protocol to wrap a final layer of security around your server.

Randy Franklin Smith

March 14, 2001

7 Min Read
ITPro Today logo

In Part 1, Part 2, and Part 3 of this series, I gave you a tour of the security settings for Windows 2000 Server Terminal Services. I pointed out how you should configure the Microsoft Management Console (MMC) Terminal Services Configuration snap-in to secure the settings for remote administration. Terminal Services is a great way to remotely administer your servers using the Internet. But even if you securely configure Terminal Services, your systems are still exposed to attack—especially to attackers who attempt to guess your administrative passwords. Here, in Part 4, I’ll show you how to use the IP Security (IPSec) protocol to wrap a final layer of security around your server.

Introducing IPSec
In general, IPSec is a great protocol because it provides packet-level integrity, authentication, and encryption. However, the real value of Win2K’s integrated implementation of IPSec is its incredible power and versatility as a tool for network security. To secure Terminal Services, you need to know how to configure IPSec so that your systems reject any unauthorized attempt to connect to Terminal Services at the network level before the attempt reaches Terminal Services. Once you've set up an authorized, secure IPSec connection between your client and your server, IPSec will encrypt all subsequent Terminal Services traffic and also inspect each packet before passing it on to Terminal Services, making sure that the packet really came from your authorized client computer and that no one modified the data in transit. In this scenario, we’ll configure a Web server to let you use your laptop to administer Terminal Services remotely over the Internet. We’ll assume that the Web server is already set up securely and that Terminal Services has been installed securely as described earlier in this series of articles. We’ll also assume that the Web server is exposed directly to the Internet, or that if a firewall is protecting the Web server, that TCP port 3389 is open. (Terminal Services uses port 3389 for communication between the client and server.)

Configuring IPSec
To set up IPSec on the server, open the MMC Local Security Policy, right-click IP Security Policies on Local Machine, and select Create IP Security Policy. When the wizard appears, click Next, enter a name for the policy (e.g., Secure Terminal Services Traffic), and click Next again. On the next page, don’t let the wizard activate the default response rule. (There's a check box that says "activate default response rule." Don't check this box.) Click Next again to arrive at the final page. Leave the Edit properties box checked, and click Finished to view the properties window (Secure Terminal Services Traffic Properties) for your new IP Security policy.

Now you need to create a rule that requires an IPSec connection before letting clients attempt to connect to port 3389. Click Add to start the IP Security Rules Wizard. Click Next on the first three pages of the rule wizard IP Security Policy Wizard until you come to the Authentication Method page, as Figure 1 shows. The IP Security Rules Wizard wants to know how the client and server computers should authenticate each other. If both the server and the client are part of the same Active Directory (AD) forest, you can select Windows 2000 default (Kerberos V5 protocol), which is relatively safe and definitely the most convenient option because there is no need to set up authentication keys. The second authentication option you can select is Use a certificate from this Certificate Authority (CA): and select one of the choices. You can use this secure option when both computers aren’t part of the same AD forest; however, this option requires additional setup for requesting and installing certificates.

To keep things simple, use this string to protect the key exchange (preshared key), which requires that you enter a string of characters both at the client and the server. The client and server both use the string as a shared secret key to encrypt and exchange dynamically created session keys whenever you connect to Terminal Services. First, type in a long string (up to 1024 bytes) of random numbers and symbols, as Figure 1 shows. Setting up IPSec this way doesn’t require that you use AD or certificates, but be aware that both the client and the server store this key in clear text in each respective registry. To help you set up the client later, copy the string to the Clipboard and save it in a text file on a floppy disk. Now click Next, which brings you to the IP Filter List page, as Figure 2 shows. You’ll need a new filter that catches connections to Terminal Services, so click Add. Change the name of the filter by typing "Incoming Terminal Services Connections," and click Add. Click Next on the first page of the Filter wizard. Change the source address to Any IP Address, and click Next. Change the destination address to My IP Address, and click Next. On the page listing the protocol types, select TCP, click Next, and select From any port. Set To this port to 3389. Click Next several times until you reach the last page, and then click Finish to produce the same results under Filters as you see in Figure 2. Close the IP Filter List window, as Figure 2 shows, to return to the IP Filter List of the IP Security Rules Wizard. Select the Incoming Terminal Services filter you just created, as Figure 3 shows.

Click Next to display the Filter Action page. So far, you’ve specified that IPSec should catch traffic on port 3389 so that IPSec will handle authentication using the preshared key you entered, but you still need to specify how IPSec will react to port 3389 connections. Click Add to create a new filter action, and click Next on the first page of the Security Rule Wizard. Select Require Encapsulated Secure Payload, as Figure 4 shows, and click Next. On the next page, you can select the filter action behavior. With IPSec you can implement a simple port-blocking firewall with the Permit and Block actions, but for our scenario, select Negotiate security, and click Next. Select Do not communicate with computers that do not support IPSec, click Next, and select High for your security method. Click Next until you reach the last page, and then click Finish. Back on the Security Rule Wizard page, select the new filter action you just created (Require Encapsulated Secure Payload), as Figure 4 shows. Click Next until you reach the last page and then click Finish. Close the Policy properties window. Back in Local Security Policy, right-click your new policy, and click Assign. Run secedit /refreshpolicy machine_policy to force Group Policy to be applied. This final action concludes configuring IPSec on your server.

Now, on your laptop, open Local Security Policy and create a new IP Security Policy, but this time let the wizard create the default response rule. Specify the same authentication method as before (preshared key). From the floppy disk, copy the key you saved earlier (the string of random numbers and symbols) and paste it in here. When you’re finished with the floppy disk, delete the file on the floppy disk to protect the key. Click Next until you reach the last page, and then click Finish. Back in Local Security Policy, right-click your new policy, and click Assign. Run secedit /refreshpolicy machine_policy to force Group Policy to be applied. Now you are finished with the client. Test your changes by opening the Terminal Services client and connecting to your server. Your laptop should connect to your server without any trouble. To make sure that your IPSec is actually functioning and protecting you, try connecting to Terminal Services from some other computer where you haven’t entered your preshared key, or simply change the IPSec key on your client by one byte, apply Group Policy again, and try to connect to Terminal Services. In both tests, the Terminal Services client should fail to open a session.

Although this has been a simple introduction to IPSec, you have many more options and opportunities if you use certificates and AD. As you can see, IPSec provides great potential for delivering rock-solid network security.

Sign up for the ITPro Today newsletter
Stay on top of the IT universe with commentary, news analysis, how-to's, and tips delivered to your inbox daily.

You May Also Like