Stop Email Server Attacks

Over the past several years, as email attacks against end users and their desktop systems have increased, direct attacks against email servers have decreased (although the decrease has been relative). However, servers are still vulnerable because attackers are still releasing exploits against Microsoft Exchange Server and even Sendmail. Let's look at a couple of common attacks and ways you can reduce or stop these attacks against your email servers.

Buffer-Overflow Exploits
A buffer overflow occurs when a software program, such as a mail server, stores more data in a data buffer than was originally allowed for and no provision exists for the unexpected input. Attackers can use this bug to make the mail server execute other programs it was never intended to execute. If the mail server runs at a privileged level, the entire system can be compromised. Even if the mail server doesn't run in privileged mode, attackers can compromise it and gain full access to its resources.

Although they can occur accidentally through programming errors, buffer overflows are a common security exploit against data integrity. In a buffer-overflow exploit scenario, the extra data can contain codes designed to trigger specific actions, such as sending new instructions to the attacked server that could damage user files, change data, or disclose confidential information.

In the past, attackers often used buffer-overflow exploits to enable the passing of worms between various servers on the Internet as well as to prove their prowess. More recently, however, buffer-overflow exploits have a more targeted purpose: They let attackers compromise a mail server so that they can then use the mail server to send spam.

This type of attack has two serious consequences. First, a compromised mail server means that attackers can read the email messages being sent to and from your company. The results can be devastating. Second, attackers can use the server resources of your company to send spam. This scenario can earn bad will for your company and violate your ISP contract, which often means termination of service.

It's important that you harden your mail servers (and any other public servers) against buffer-overflow exploits and other types of attacks. You can also take several other protective measures.

Server Hardening
The best way to reduce the chance of a mail server compromise is to harden the mail server itself. In all situations, hardening is a worthwhile effort. On hardened servers, especially Internetfacing servers, fewer services are available for exploitation and those services are generally "compartmentalized." The following measures are generally required for hardening:

By hardening servers, you can dramatically reduce their vulnerability. Unfortunately, hardening mail servers often isn't enough. A better solution is to both harden the server and provide additional filtering for email traffic before it actually hits the server. You can filter email traffic early by using network appliances, managed services, and software integrated into an existing mail system (e.g., Microsoft Exchange). Keep in mind that you want to layer your defenses—for example, by hardening internal mail servers while at the same time deploying vendor-hardened network appliances to protect the perimeter.

Network Appliances
Mail-filtering network appliances are deployed in front of internal mail servers. These appliances usually provide two types of firewalls: a packet-filtering firewall and an application-level firewall. As a packet-filtering firewall, a network appliance allows only valid TCP/IP traffic to ports that mail services use (e.g., SMTP, often POP3 and IMAP). As an application-firewall, the appliance ensures that the sending server properly uses SMTP and follows relevant IEEE Requests for Comments (RFCs) and common practices (e.g., having reverse DNS set up).

Network appliances tend not to be susceptible to attack for several reasons. First, most appliances run on heavily customized OSs. These OSs have been stripped of most extra services that would let attackers gain a foothold on the system (or the OS has been designed from scratch specifically for the appliance). Second, engineers typically follow best practices when hardening the appliance. Finally, an appliance permits only a limited set of traffic (i.e., traffic related to mail transport) to and from the mail server and even that traffic is carefully scrutinized.

Figure 1 shows a network appliance located in front of an internal mail server. This placement lets the appliance protect internal servers and lets you offload processing from the internal mail servers to the appliance.

Managed Services
With a managed service, all email messages are sent first to an offsite service that filters email, as Figure 2, shows. This service then forwards valid email messages to your organization's mail server.

For this strategy to be effective against direct attacks that use mail protocols, the internal mail server must not accept any connections other than those that the managed service initiates. However, such services work for inbound email traffic only. Outbound traffic is still sent directly to other servers on the Inter-net, enabling possible exploits that use mail protocols (e.g., a receiving mail server exploits a buffer-overflow vulnerability in your sending mail server's software during the SMTP transaction).

Integrated Software
Finally, you can install integrated software to help protect your mail server. This locally installed software hardens servers against network attacks. Often, integrated software works at the application layer (i.e., SMTP) to protect a server from exploits. Some integrated software replaces a server's local TCP/IP stack with a customized hardened version. More often, however, local filtering software works in conjunction with the mail software rather than creating a wall between it and external systems. Integrated software that takes this approach can help if an attacker has direct access to the mail server (e.g., if a trusted internal user initiates the attack).

DoS Attacks and DHAs
Denial of Service (DoS) attacks reduce the ability of the target system to function. In the case of an email server, attackers intend to slow down or crash the server. Attackers perpetrate DoS attacks in several ways, including by consuming network resources and by launching directory harvest attacks (DHAs).

When attackers perform DoS exploits through network resource consumption, the attacks often focus on consuming all the available incoming connections of the target machine. Because SMTP is a TCP protocol, a successful exploit requires only that the attackers request more TCP connections than are available. That is, attackers create more connections to the mail server than the mail server can handle. The mail server is then no longer able to accept valid incoming connections from legitimate mail servers.

You'll find few mail server—based solutions that prevent DoS attacks. Most mail servers run on general-purpose OSs that aren't geared to prevent DoS attacks. Even on a hardened UNIX system, various network settings are required to increase the server's ability to withstand a large DoS attack. Therefore, organizations generally purchase systems specifically created to detect and deter DoS attacks or hardened filtering appliances that can accept many more concurrent connections than a general-purpose mail server. Such filtering devices often can better detect DoS attacks and take defensive measures.

DHAs are extremely resource-intensive attacks by spammers to determine valid addresses to use for future spamming. In a DHA, the mail server load can increase dramatically, affecting delivery of valid email messages. Additionally, the local mail server will attempt to return nonde-livery reports (NDRs) for invalid addresses to the From addresses that spammers use. Returning NDRs generates additional outgoing email traffic, consuming expensive bandwidth and further increasing the load on the mail server. And because most From addresses that spammers use are spoofed, the NDR delivery often times out, requiring the mail server to attempt delivery at a later time. All in all, DHAs are an expensive form of attack against mail servers.

Unfortunately, you'll find few ways to mitigate the risk of a DHA. One solution is the use of managed services. Managed services generally maintain many more mail servers than an organization can provide; therefore, DHAs don't affect the over-all mail delivery as much. Another solution is to install a front-end filtering appliance optimized against this type of attack. A list of legitimate email users is maintained on the appliance (through a static list or Lightweight Directory Access Protocol—LDAP—access to an internal directory) so that the filter doesn't relay any email messages addressed to invalid users to the internal mail server. Additionally, the filtering appliance won't generate NDRs, so a DHA won't create an additional outbound traffic burden.

Editor’s note: This article is excerpted from Spam Fighting and Email Security for the 21st Century (Windows IT Pro eBooks). The book is available at http://www.windowsitpro.com/eBooks.