Toolbox: Encrypt Telnet via IPsec in 10 Steps

As ubiquitous as Telnet is, many security and IT administrators shun it because Telnet traffic passing between the client and server is unencrypted and visible to any network snooper who might be listening. Before you resort to cobbling together a Win32 port of OpenSSH (a freeware version of the Secure Shell—SSH—suite of network-connectivity tools) or spending thousands of dollars for a commercial version of SSH, consider using IPsec to encrypt your Telnet traffic. Together, Telnet and IPsec provide a secure and relatively easy-to-deploy solution for remote-console access for Windows Server 2003 servers. Although configuring IPsec can be complicated, here I provide 10 quick, straightforward steps to follow for encrypting Telnet over IPsec. You can also extend these examples for encrypting other network traffic to suit your own security needs.

A Few Preliminaries
The sample configuration that I'll walk you through uses Kerberos—the Windows IPsec default authentication method—which means that your client and server must reside in the same or trusted domains. IPsec supports alternative authentication mechanisms (e.g., certificates) that you can use if Kerberos isn't an option for your domain. However, such mechanisms can be trickier to set up and require an existing public key infrastructure (PKI).

When you configure IPsec for the first time, the IPsec wizards might seem daunting. One wizard invokes another wizard, which invokes yet another. But after you've created the objects once, the interaction of the elements will make sense. To encrypt Telnet traffic over IPsec, you'll create these objects:

Windows 2003 includes helpful tools for troubleshooting failed IPsec connections. One such tool is the Microsoft Management Console (MMC) IP Security Monitor snap-in, which shows you the filters, policies, and security associations for both Main Mode and Quick Mode IPsec. Also, you can use a network sniffer to watch network traffic as the IPsec connection between the client and server is negotiated. Although the sniffer doesn't display the contents of the encrypted traffic, you can see the client and server setting up the IPsec connection via Internet Security Association and Key Management Protocol (ISAKMP, TCP port 500).

For more detailed information about configuring IPsec, see "IPSec and Group Policy: The Next Step," September 2002, InstantDoc ID 26112; "IPSec and Group Policy: A Stronger Defense," August 2002, InstantDoc ID 25730; and the Microsoft TechNet IPSec Troubleshooting site(http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/serverhelp/b0b726fc-c6b7-426a-964f-9c7b1c8ef3a8.mspx).

Create the IPsec Policy
To encrypt Telnet traffic for your domain, follow these steps:

1. On each server that you want to manage by using Telnet, open the Services applet in Administrative Tools and enable the Telnet service to start up automatically, then start the service.

2. Create a new Group Policy Object (GPO) and link it to the organizational unit (OU) containing the computers (clients and servers) on which you want to require encrypted Telnet sessions. If you choose, you can link this GPO at the domain level for every computer in the domain to encrypt the Telnet sessions.

3. Edit this new GPO by using Group Policy Editor (GPE). Navigate to Computer Configuration, Windows Settings, Security Settings. Right-click IP Security Policies on Active Directory (domain) and click Create IP Security Policy to launch the IP Security Policy Wizard.

4. Walk through the wizard by following these steps:

Create the Security Rule and IP Filter List

5. Next you'll add a new IP Security rule that encrypts Telnet traffic. Create the rule by clicking the Rules tab of your newly created policy's properties dialog box. Click Add to start the Security Rule Wizard.

6. Proceed through the Security Rule Wizard as follows:

7. The IP filter list defines the protocol and addresses that the IPsec rule uses. In the New IP Filter List dialog box that's displayed, name the new IP filter list (e.g., Telnet) and click Add to run the IP Filter Wizard. Proceed through the wizard by entering or selecting the following information:

8. Click OK to close the new IP Filter List dialog box, which returns you to the Security Rule Wizard in step 6. Click the radio button next to your newly created IP filter list (e.g., Telnet) and click Next.

Create the Filter Action

9. Now we need to create the filter action. To do so, click Add, which launches the Filter Action Wizard. Proceed through the wizard as follows:

Accepting unsecured communications is necessary because when a client computer initiates a Telnet session, it first tries to do so by using TCP port 23. The IPsec-protected server needs to recognize and accept this unsecured request but then immediately respond by setting up IPsec communications. Without this security negotiation, the client can't connect to the server. The benefit of this filter-action configuration is that it lets your clients still connect to routers, switches, and other non-IPsec­enabled Telnet servers if they need to.

10. Exiting the Filter Action properties dialog box and wizard returns you to the Security Rule Wizard. Click the radio button next to your newly created filter action (e.g., Encrypt Data) and click Next. Leave the authentication method for your new rule at the default of Active Directory (i.e., Kerberos V5 protocol), then exit the wizard.

You've now configured the core components of an IPsec policy. You can review the details of this policy by viewing the policy's properties pages, which Figure 1 shows, or click OK twice to exit the policy's properties.

Assign the IPsec Policy ... and You're Done!
You're now ready to assign the policy. Return to GPE and select IP Security Policies on Active Directory (domain). In the right pane, right-click your new IPsec Telnet policy and select Assign. Force the GPO update by running the Gpupdate command on each computer running the Telnet server (or you can wait for the GPO to replicate). You must run Gpupdate on both the client and server.

IPsec used in conjunction with Telnet dramatically improves Telnet security. Running Telnet over IPsec might not be ideal for every organization because of the additional requirement and complexity of setting up an Active Directory (AD) GPO. But if you manage a number of Windows servers and avoided Telnet in the past because of its nonsecurity, IPsec lets you consider it as another method to support remote administration of your servers.