Configuring ISA Server for SSL/TLS

More and more organizations are adopting Microsoft Internet Security and Acceleration (ISA) Server 2000 as a proxy and firewall solution for their IT infrastructure. ISA Server is the follow-up product to Microsoft Proxy Server 2.0, but unlike its predecessor, which was primarily a Web caching solution, ISA Server can serve as a full-blown firewall. In addition, ISA Server provides advanced packet filtering and application-proxy firewall functions. The firewall also uses a stateful engine, which secures Secure Network Address Translation (SNAT). For a complete overview of ISA Server's features, visit the Microsoft ISA Server Web site at http://www.microsoft.com/isaserver, visit the ISAserver.org Web site at http://www.isaserver.org, or see "Related Reading," page 2.

Organizations can use ISA Server as a proxy, a firewall, or both. In this combined role, ISA Server protects your Web servers from the dangers of a public network such as the Internet and provides Web content caching for Web users (i.e., proxying) and Web servers (i.e., reverse proxying). Whereas vendors such as Cisco Systems and Nokia provide dedicated hardware-based firewall and proxy solutions equipped with HTTP proxy software, ISA Server is simply a software-based solution.

Let's look at how to set up the Secure Sockets Layer (SSL)/Transport Layer Security (TLS) protocols for secure Web connections. Specifically, I review which SSL scenarios you can use with HTTP proxies, how to request and obtain SSL certificates, and how to configure ISA Server for SSL bridging and tunneling. I also mention a key factor when securing ISA Server: the SSL processing load.

The ISA Server installation I describe in this article uses ISA Server Service Pack 1 (SP1), which requires Windows 2000 SP2, and ISA Server Feature Pack 1. Microsoft released the feature pack earlier this year, and you can download it at http://www.microsoft.com/isaserver/featurepack1/default.asp.

Securing an OWA Environment
Figure 1, page 2, shows a typical Microsoft Exchange 2000 Server Outlook Web Access (OWA) setup. OWA lets Exchange users access their mailbox data from an Internet browser. In this environment, ISA Server combines a firewall with a proxy solution and functions as an advanced software-based HTTP proxy server. Instead of communicating directly with the OWA front-end Web server, Web clients connect through the HTTP proxy server, which greatly enhances the overall security of the OWA infrastructure. ISA Server also caches and analyzes the OWA Web content.

From a perimeter security point-of-view, the OWA users are on an untrusted network (e.g., the Internet) and the OWA front-end Web server and Exchange server are in a trusted network zone (e.g., an internal corporate network). ISA Server separates the untrusted security zone from the trusted security zone. Although you can take several other steps (including adding a demilitarized zone—DMZ) to securely deploy ISA Server, these permutations are beyond the scope of this article.

The Crucial Role of SSL/TLS
Web sites that handle confidential information (e.g., credit card numbers) often use SSL to secure Web traffic. SSL is a security protocol operating at the transport layer of the Open System Interconnection (OSI) networking stack. SSL provides authentication, confidentiality, and integrity protection to application-level protocols (e.g., HTTP, SMTP). Netscape developed SSL, and you can download the most recent SSL specification (SSL 3.0) at http://wp.netscape.com/eng/ssl3. In 1999, Internet Engineering Task Force (IETF) Request for Comments (RFC) 2246 standardized SSL under the name Transport Layer Security (TLS) protocol (more information is available at http://www.ietf.org/rfc/rfc2246.txt). TLS is also known as SSL 3.1. In the discussion that follows, the term SSL refers to both the SSL and TLS protocols. Eric Rescorla's SSL and TLS: Designing and Building Secure Systems (Addison-Wesley, 2001) reveals all the nuts and bolts of both protocols.

SSL and HTTP Proxies
When working with HTTP proxies, you can configure SSL in one of two ways: SSL tunneling or SSL bridging. Web Table 1 (http://www.secadministrator.com, InstantDoc ID 38099) shows the different SSL approaches and their advantages and disadvantages.

SSL tunneling provides true end-to-end SSL protection, as Figure 2 shows. The SSL tunnel starts on the browser and ends on the Web server. SSL tunneling requires an SSL authentication certificate on the Web server; an SSL client certificate on the client is optional. The primary disadvantage of using SSL tunneling is that it breaks the security role of an HTTP proxy, which typically inspects HTTP request content before letting the request through to the Web server. When you use SSL, the HTTP proxy can't inspect the HTTP request because the SSL tunnel encrypts the HTTP content. Because SSL operates on the transport level of the OSI networking stack and HTTP operates on the application level, the SSL tunnel is always in place before the HTTP application-level traffic reaches the destination host.

To configure SSL tunnels to flow through HTTP proxies, SSL-enabled applications use the HTTP CONNECT method. This method tells the HTTP proxy to ignore the content of an SSL session and to simply forward the SSL packets to the destination host (in our example, an OWA front-end Web server). RFC 2817 defines the HTTP CONNECT method.

SSL bridging, unlike SSL tunneling, lets you add a secure pathway without removing the HTTP proxy security because the SSL tunnel starts or terminates on the HTTP proxy. You can set up SSL bridging in one of three ways:

Of these approaches, only SSL bridging options 1 and 3 are truly secure. From a security standpoint, SSL bridging option 2, which doesn't secure traffic between the client and the HTTP proxy (i.e., traffic that typically flows across a public network), doesn't make sense. This option secures traffic only between the HTTP proxy and the Web server (i.e., traffic that typically flows across a trusted network or DMZ).

Let's take a closer look at how to configure ISA Server to support SSL bridging and SSL tunneling. Of these two options, SSL bridging is the most commonly used approach, so most of the discussion focuses on that option.

Obtaining an SSL Certificate for the ISA Server
Before you can configure ISA Server for SSL bridging option 1 or 3, you must install an SSL server certificate on the computer running ISA Server. In my test environment, I set up a Win2K enterprise Certificate Authority (CA) in a trusted security zone (i.e., the intranet) to generate SSL server certificates. In environments lacking internal CAs, you must obtain SSL server certificates from a public CA such as VeriSign.

In the following discussion, I explain only how to obtain an SSL certificate from ISA Server—describing how to make sure that the ISA/OWA users trust the SSL certificates of the issuing CA is beyond the scope of this article. The ISA Server Feature Pack 1 documentation explains these instructions.

To request an SSL server certificate for ISA Server, start the Microsoft Management Console (MMC) Internet Services Manager (ISM) snap-in on the CA server, then open the default Web site properties. From the Directory Security tab, click Server Certificate to open the Certificate Request Wizard. Select the Create a new certificate option. In this example, you can also select Send the request immediately to an online certification authority option because the CA is an enterprise CA that's published in Active Directory (AD) and thus visible to the Web server. Make sure you complete the DNS name (e.g., OWA.mycorp.com) for your OWA front-end Web server in the Common Name field. After you successfully create the SSL server certificate, you must export the certificate and the corresponding private key from the CA server and import them on the ISA Server system.

The main reason the Web server provides ISA Server's SSL certificate is ease of configuration—IIS comes with an SSL certificate request wizard. You can also launch the SSL certificate request from ISA Server, in which case ISA Server's private key never leaves the machine, but the certificate request procedure is more complex.

To export the SSL server certificate and the corresponding private key from the Web server, open the default Web site properties and go once more to the Directory Security tab. This time, click View Certificate. In the Details tab, click Copy to File to open the Certificate Export Wizard. Select the Yes, export the private key option. In the Public Key Cryptography Standard #12 (PKCS#12) export file format options, select the Enable strong protection option (so that the wizard will prompt you to enter a password) and the Delete the private key if export is successful option. Save the PKCS#12 formatted file, then copy the file to the ISA Server system.

To import the SSL server certificate and corresponding private key into ISA Server's certificate and private key store, double-click the PKCS#12 file from Windows Explorer to open the Certificate Import Wizard. Leave all the defaults set in the wizard and make sure you enter the correct password. The wizard will automatically store the certificate in ISA Server's local machine certificate store.

Next, you must configure ISA Server to use the SSL server certificate. Start the MMC ISA Management snap-in, locate your ISA Server object, right-click the object, then select Properties. Select the Incoming Web Requests tab. Select Use the same listener configuration for all IP addresses or Configure listeners individually per IP address, depending on your ISA Server configuration and which connections you want to secure with SSL. Highlight the appropriate server from the list, then click Edit to open the Add/Edit Listeners dialog box. Select the Use a server certificate to authenticate to web clients check box. Click Select, then select the correct certificate (in this example, the one issued to OWA.mycorp.com).

Configuring ISA Server for SSL Bridging
After you create the SSL certificate, you're ready to configure SSL bridging. The first step is to ensure that ISA Server accepts incoming SSL requests. To verify that ISA Server's SSL listener is enabled, select the Enable SSL listeners check box on the Incoming Web Requests tab of the ISA Server object's Properties dialog box. By default, ISA Server's listener is disabled; however, when enabled, ISA Server listens on port 443 for incoming SSL requests.

Next, you must create an ISA Server destination set and Web-publishing rule for OWA. A destination set is a group of servers for which ISA Server will perform specific actions. In our example, we want ISA Server to perform SSL bridging when an HTTP Secure (HTTPS) request comes in for the OWA Web pages. To create a destination set for OWA, expand the Policy Elements container in the ISA Management snap-in, right-click the Destination Sets container, then select New, Set to open the New Destination Set dialog box. Give the set a name (e.g., OWA), then click Add to start adding OWA destinations. In this example, the OWA front-end Web server's DNS name, which is published to OWA users, is OWA.mycorp.com. Because OWA requires a destination set for three Web paths (/exchweb/*, /public/*, /exchange/*), you'll need to add these paths. For each path, complete the destination DNS name (in this case, OWA.mycorp.com) and the path.

To redirect all HTTP requests for OWA.mycorp.com to the OWA front-end Web server with IP address 10.0.0.45, you need to create an OWA Web-publishing rule. To create the rule, expand the Publishing container in the ISA Management snap-in, right-click the Web Publishing Rules container, then select New, Rule to open the New Web Publishing Wizard. Give the rule a name (e.g., OWA), then apply the rule to the OWA destination set you just created. Apply the rule to any request, and tell the rule to redirect the request to the OWA front-end Web server (e.g., the OWA front-end Web server's IP address—10.0.0.45). Also, make sure that you select the Send the original host header to the publishing server instead of the actual one (specified above) check box.

Next, if you're setting up SSL bridging option 1, you need to configure the OWA Web-publishing rule's bridging properties to ensure that the SSL tunnel terminates at the ISA Server HTTP proxy and that no new SSL tunnel initiates to the OWA front-end Web server. Select the Bridging tab in the rule's properties dialog box, then select the HTTP requests radio button in the Redirect SSL requests as box.

To make the SSL bridging option 1 setup work with OWA, you must add a setting to the registry on the ISA Server system. (This registry hack works only with ISA Server SP1 and later.) Navigate to the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesW3ProxyParameters registry subkey. From the Edit menu, select New, DWORD Value; name the value AddFrontEndHttpsHeader; then set it to 1. The Microsoft article "Secure OWA Publishing Behind ISA Server May Require Custom HTTP Header" (http://support.microsoft.com/?kbid=307347) provides additional information about this registry setting. To make sure that your ISA Server configuration changes go into effect, stop and restart ISA Server's Web Proxy and Firewall services.

You can also use the Bridging tab of the OWA Properties dialog box to configure ISA Server for SSL bridging options 2 and 3. To add an SSL tunnel for unsecured HTTP requests (SSL bridging option 2), select the SSL requests (establish a secure channel to the site) radio button in the Redirect HTTP requests as box. To add an SSL tunnel between the HTTP proxy and the OWA front-end Web server (SSL bridging option 3), select the SSL requests (establish a new secure channel to the site) radio button in the Redirect SSL requests as box. Keep in mind that both bridging options require an SSL certificate on the OWA front-end Web server.

The Bridging tab contains two other important SSL-related configuration options. If you want ISA Server to connect only to the published Web site on condition that the user types an HTTPS URL in the browser address bar, select the Require secure channel (SSL) for published site check box (this setting applies only to SSL bridging options 1 and 3). If you want ISA Server to perform a strong SSL certificate-based authentication to the OWA front-end Web server, select the Use a certificate to authenticate to the SSL Web server check box and select a certificate (this setting applies only to SSL bridging options 2 and 3). In the latter case, you should install ISA Server's SSL client-side authentication certificate in the W3Proxy service's certificate store on the ISA Server system.

To help facilitate the configuration of SSL bridging in an OWA environment, Microsoft has provided an OWA Publishing Wizard in ISA Server Feature Pack 1, which is available for download at http://www.microsoft.com/downloads/details.aspx?familyid=2f92b02c-ac49-44df-af6c-5be084b345f9&displaylang=en. The new wizard, which Figure 6 shows, is accessible from the ISA Web Publishing Rules container—simply right-click the container, then select New, Publish Outlook Web Access Server. The wizard guides you through the different configuration options and automatically adds the AddFrontEndHttpsHeader registry setting.

Configuring ISA Server for SSL Tunneling
To configure ISA Server for SSL tunneling, you need to use ISA Server's Server Publishing feature. Compared with configuring SSL bridging, setting up SSL tunneling is easy because you don't need to create an SSL certificate for ISA Server.

Start by making sure that ISA Server's SSL listener is disabled. The SSL listener doesn't need to be enabled because it's used only for ISA Server's Web-publishing function. To make sure the SSL listener is disabled, clear the Enable SSL listeners check box on the Incoming Web Requests tab of the ISA Server object's Properties dialog box.

Next, create a server-publishing rule for your OWA front-end Web server. From the ISA Management snap-in, expand the publishing container, then right-click the Server Publishing Rules container. Select New, Rule to open the New Server Publishing Rule Wizard, as Figure 7 shows. Type a name for the rule (e.g., OWA), and click Next. On the Address Mapping page, type the IP address of your OWA front-end Web server in the IP address of internal server box, then type the IP address from which your ISA Server is accessible to the outside world in the External IP address on ISA Server box. On the Protocol Settings page, select the HTTPS Server protocol. On the Client Type page, select Any request. To make sure that your ISA Server configuration changes go into effect, stop and restart ISA Server's Web Proxy and Firewall services.

Don't Forget the SSL Processing Load
Throughout this article, I've discussed how to configure ISA Server to support various SSL tunneling and bridging flavors, but I haven't addressed how SSL processing affects ISA Server performance. These performance considerations are especially important if you're setting up SSL bridging option 1 or 3 in large OWA environments (see Web Table 1 for a more detailed explanation of the different SSL bridging options). In either situation, you might want to evaluate SSL crypto-accelerator cards or dedicated SSL-processing hardware appliances to help offload SSL processing from ISA Server.