Dynamic Blacklists Demystified

These days, using mail-filtering software to help eliminate unwanted email is a necessity. Mail-filtering solutions use a variety of techniques to filter both wanted and unwanted mail and can be desktop-based, server-based, or a combination of both. One filtering type that I've found to be helpful in reducing the amount of spam that comes into my inboxes is the dynamic blacklist. Although dynamic blacklist services are somewhat controversial, in my experience they work well and deserve a place in every spam fighter's arsenal. The trick to getting the most out of dynamic blacklists is to understand both how to work with them and how to choose a blacklist service provider wisely.

Understanding Dynamic Blacklists
Most users are familiar with static blacklists. These are lists of email addresses and domains from which a mail server will block incoming mail. Many software mail-filtering solutions let users either add domain and email addresses to the server's blacklist or maintain their own client-based blacklist. Static blacklists are typically updated manually.

Dynamic blacklists rely on blacklist service providers, which track IP addresses (and sometimes domain names) that spammers are known to use. Mail filters that support blacklist services can query service providers about a given IP address, including the address of the message sender and any addresses that relayed a particular message along its delivery route. If the result of the query shows that the IP address is on the service provider's blacklist, then the probability is high that the message is spam. Some blacklist service providers also track addresses that are known to send viruses, Trojans, worms, back doors, and other sorts of malware. Those providers' blacklists can be useful in helping you keep such nuisances out of your network.

Working with Dynamic Blacklists
Dynamic blacklists rely on DNS lookups to query for a given IP address (or, in some cases, a domain name) that is either a delivering system's address or the addresses listed in a message's Received: headers. To query a given blacklist service, a user's mail-filtering software must build a query to send to the service provider's DNS servers. Queries are typically structured as PTR records that query for an A record. When a blacklist filter receives a positive response to a query, the likelihood is high that the delivering system (or the address in question when a query targets the addresses in a message's Received: headers) is used for sending spam. Upon receiving a positive response to a query, the blacklist filter can either deny receipt of messages from the identified system, delete the received messages, place the messages in a queue for further review, or assign a weight designation to the messages that can be used in conjunction with the weighting systems in a user's personal mail-filtering solution.

You can structure a query yourself by using a standard DNS lookup client such as nslookup from a Windows command line to perform ad hoc blacklist checking. You don't have to query the blacklist service's DNS server directly. When you query your own DNS servers, they will pass the query to the blacklist service as designated in the PTR query. For example, to query the blacklist service provider Spamhaus.org to determine whether the address 198.0.0.1 is known to be used by spammers, you would structure your query as follows:

1.0.0.198.spamhaus.org

Query responses from a given blacklist service provider's DNS servers vary depending on the provider's particular specification, but in general, blacklist services that return a positive query result typically designate an address in the 127.0.0.x address space. (However, blacklist services will never return the address 127.0.0.1 because that address is always reserved as the "localhost" address on all systems running TCP/IP protocols.) A query will fail to return results if the queried address isn't in the particular blacklist service provider's databases. By adhering to this standardized query-response format, developers can more easily create a common blacklist service filter that supports most blacklist service providers' specifications.

Choosing an Effective Blacklist Service Provider
Now that you've mastered the basics of how dynamic blacklist services work, the next step is to choose service providers carefully. Many blacklist service providers exist, and figuring out which are most effective can be hard to determine. To help you start, I'll discuss a few blacklist services that I've tried and found to be reasonably trustworthy.

Keep in mind that it isn't a good idea to delete messages based solely on the results of a query to a blacklist service provider. It's better to instead use query results as part of an overall strategy for determining the probability that email messages are spam. Legitimate mail can originate from the same IP addresses that spammers use, and positive query results don't recognize this. You'll lose less legitimate email if you don't automatically delete messages that generate positive query results.

Blacklist service providers vary in which services they provide, so take time to investigate what a given provider can do. For example, in my testing of mail-filter software, I've found that mail filters that use dynamic blacklists should query every mail server address found in a message's Received: headers. Sure, doing so requires more processing time per message, but it also significantly increases the likelihood of detecting spam. Why? Because a spammer might hijack any number of systems to relay mail. Therefore, the connecting IP address that delivers the message to the destination mail server might not be in a blacklist service provider's database, yet other addresses listed in the headers might be. Some blacklist service mail filters don't query all the Received: headers, or query only for the delivering system's IP address.

Blacklist Service Providers: A Selected List
Spamhaus (http://www.spamhaus.org) is a well-known and popular blacklist service that provides query results that reveal sources of spam as well as sources of exploits such as viruses, Trojans, and worms. The company also provides the Registry of Known Spam Operations (ROKSO) that provides information about known spammers, whether they are individuals who bulk mail on behalf of advertisers or advertisers who don't conform to opt-in or opt-out policies. You can query Spamhaus's blacklists by using sbl-xbl.spamhaus.org as the domain suffix for your queries. To view the ROKSO database, go to http://www.spamhaus.org/rokso/index.lasso.

When you query Spamhaus's DNS servers for a given address, if your query returns an address of 127.0.0.2, the queried address is known to Spamhaus as a source of spam. If the query returns an address in the range between 127.0.0.4 and 127.0.0.6, the queried address is known as a source of exploits such as viruses, Trojans, and worms. Spamhaus is unique in its ability to report on known sources of exploits.

SpamCop (http://www.spamcop.net) is another well-known and popular blacklist service provider. By its own admission, the SpamCop Blocking List (SCBL) is "an aggressive spam-fighting tool." By using this list, you can block a lot of spam—and you can also block or filter wanted email. Because of this limitation, strongly consider using the SCBL as part of a mail scoring system and add legitimate email senders to a whitelist that you establish as part of your overall spam-fighting strategy. You can query SpamCop's blacklists by using bl.spamcop.net as the domain suffix for your queries.

Spam and Open Relay Blocking System (SORBS—http://www.dnsbl.us.sorbs.net) lists approximately 3 million open relays, open proxies, dynamic IP address space, and otherwise compromised host systems that spammers use. Listing dynamic IP address space is controversial and a highly suspect practice. The justification for such listing is that legitimate mail servers should have a fixed IP address. However, many people prefer to use their own mail servers for outbound mail and to do so must use dynamically assigned IP addresses. Placing dynamic IP address space in its blacklist database is damaging to innocent individuals and reflects a dangerously low tolerance at SORBS for the selective needs of law-abiding individuals. I caution against using SORBS for anything other than spam-probability weighting; otherwise, you're bound to lose legitimate email. You can query the SORBS blacklists by using dnsbl.sorbs.net as the domain suffix for your queries.

Composite Blocking List (CBL—http://cbl.abuseat.org) uses spam traps set around the Internet to identify IP addresses that run suspected open proxies (whether via HTTP, SOCKS, AnalogX, Wingate, or other protocols) that are known to have been used to send spam or malicious code. If you use Spamhaus, then the CBL blacklist will be included in your queries, and in that case, using CBL as a separate blacklist service will be redundant. However, if you choose not to use Spamhaus, you can use the CBL on its own. You can query the CBL blacklists by using cbl.abuseat.org as the domain suffix for your queries.

Blitzed Open Proxy Monitor List is also included in queries to Spamhaus but can be used as a separate service. The service lists open proxies of various types that are known to have been used to send spam. The service detects open proxies by using spam email information and by performing checks against machines that connect to Internet Relay Chat (IRC) systems. You can query the Blitzed Open Proxy Monitor List blacklists by using opm.blitzed.org as the domain suffix for your queries.

Dnsbl.net.au (http://www.dnsbl.net.au/) has no formal name other than its domain name. The service provides a very large aggregate blacklist compiled from many sources and as a result produces a lot of positive results for queried IP addresses. The service tracks open proxies, open mail relays, and lots of other data that you can view on the site's Status page. This service also lists a lot of dynamic IP address space. You can view the Web site home page and query the aggregate blacklist by using t1.dnsbl.net.au as the domain suffix for your queries. You can also selectively query more than 20 categorized blacklists, the DNS addresses of which are listed on the Status page.

510 Software Group (http://www.five-ten-sg.com/blackhole.php) is less well known than the other blacklist service providers I've described. The service provides query results for spam sources, bulk mailing services that don't use opt-in policies, open relays, the end points of multistage open relay networks, Web sites that run mailing scripts prone to abuse by spammers, and sources of malicious code. The service also lists dynamic IP address space as well as network addresses of ISPs who allegedly refuse to remove spammers from their networks. You can query the 510 Software Group blacklists by using blackholes.five-ten-sg.com as the domain suffix for your queries.

SURBL—Spam URI Realtime Blocklists (http://www.surbl.org) is unique in that it lets you check domain names, rather than IP addresses. You can harvest domain names from URLs in the headers and body of a message and check to see whether SURBL lists them. If it does, the chances are good that the message is from a known spammer. To query SURBL, you must extract the domain name from a URL or message header and remove any host names. Then, you structure a query with surbl.org as the suffix, and query for an A record: for example, domain.tld.multi.surbl.org, where domain.tld is the domain name in question. If you receive a response with an address in the 127.0.0.x range, then the message is likely from a known spammer.

Evaluating Blacklist Service Provider Performance
If you search the Internet, you'll find dozens of blacklist service providers. Evaluating providers is difficult to do without reading lots of reports that are posted on the Internet by various users, and doing so is well worth your time. Fortunately, the task of gauging the effectiveness of the results that particular service providers offer is much easier. Jeff Makey maintains a Web site at http://www.sdsc.edu/~jeff/spam that offers helpful information. In particular, Makey publishes the results of his weekly surveys of various blacklist services. He bases his survey queries on the connecting IP addresses that deliver mail to his network, and he queries blacklist services to determine whether they list particular IP addresses. The survey results show which services offer the most accurate reports according to his particular queries. Obviously, query results will vary; nevertheless, Makey's reports are helpful as a gauge of how useful a given service might be for individual users.

Keep in mind that any blacklist provider listed in Makey's surveys might or might not suit your needs today, and how well a particular provider works for you can change depending on the type of spam you receive. Spammers tend to be moving targets, particularly when they use hijacked systems to create robot mailer networks. You also need to consider the influence of false positives, which can cause you to lose legitimate email depending on how your particular mail-filtering solution uses blacklist data. I can't stress enough the importance of thoroughly testing any service you use to determine whether it satisfies your needs.

The blacklist service providers that garner top results in Makey's reports are worth trying as part of your own mail-filtering solution. The service providers that I've listed above are among the top performers from his survey results of November 2004. If you already use or intend to use blacklist services, be sure to check out Makey's reports and bookmark his site for periodic review—it's an excellent resource.

The Lowdown
Querying dynamic blacklists increases an email message's processing overhead, and DNS-lookup lag times can slow down mail delivery, especially on networks that receive huge amounts of mail. Some blacklist service providers offer zone transfer service as a feature. If your network receives tens of thousands of messages or more on a daily basis, consider using DNS zone transfers to download a particular service provider's blacklists to your network. Zone transfers let you perform DNS queries locally, which dramatically increases mail-processing speed and significantly reduces your network traffic.

Using dynamic blacklist services can be controversial; in particular, many users complain that some services will blacklist IP addresses without much, if any, investigation. In my experience, blacklist service providers such as the ones I've listed here work fairly well, even if they are occasionally stubborn in their policies about which networks they list or remove from their lists. As with all types of services, users need to exercise due diligence and choose service providers wisely.