Assessing Vulnerability Assessment Tools

Computer security professionals know that to defeat malicious intruders, you need to know how to attack like one. Intruders spend much of their time searching for systems with known vulnerabilities: All they need is patience and a chunk of exploit code to succeed in cracking a system. They use Ping or some other utility to locate potential victim machines by IP address or domain name. Then, they find out which OS and applications the hosts are running and run the related exploit code. Or, an intruder's worm creation can whip across the Internet, knocking on every door and working its tricks against every machine without even trying to find out whether the host contains the necessary software for the exploit to succeed. If the SQL Slammer (aka Sapphire) worm is any indication, this cracker strategy succeeds.

Vulnerability assessment tools automate the cracker exploration process and let network administrators assess the security readiness of their networks. Security policies, ACLs, and signed user agreements mean little if your systems are full of exploitable holes. If you can find the holes before a malicious intruder can, and close them, you've gone a long way toward making your network safer. Let's discuss vulnerability assessment tools in general; popular vulnerability assessment tools for Windows systems, which Table 1, page 34, lists; and trends in protecting against intrusion.

The vulnerability assessment tool market comprised only a few major players 2 years ago but now includes more than 40 vendors. A few products have come and gone, and some of the major network security vendors have abandoned their initial efforts because consumer demand for vulnerability assessment tools didn't meet expectations. Although host-based vulnerability assessment tools are still the most popular products, the greatest vendor growth has been in specialty scanners, such as those that scan Microsoft SQL Server databases, Web servers, and wireless LANs (WLANs). Most vulnerability assessment tools fall into one of a few different categories: host-based, application-layer (database or Web), and password and account checkers.

Host-Based Vulnerability Assessment Tools
When people think about vulnerability assessment tools, they usually have host-based tools in mind. A host-based vulnerability assessment tool finds and identifies the OS running on a particular host computer and tests it for known deficiencies. A host tool can tell the difference between a Windows 2000 system and a UNIX box and test accordingly. Most of these tools will look for and test common applications and services on each platform. For example, if a vulnerability assessment tool finds a UNIX host, it might test for daemons, Sendmail, or Samba shares. If a tool finds a Win2K host, it might test for Microsoft IIS, open NetBIOS shares, and search for weak passwords.

A Windows-based vulnerability assessment tool should understand the operational differences among different Windows versions. For example, testing IIS exploits or remote procedure call (RPC) Denial of Service (DoS) attacks against a Windows 98 machine is pointless. Several popular vulnerability assessment tools have UNIX roots: They excel at testing UNIX and Linux systems and test Windows systems as a byproduct. If your network contains nothing but Windows machines, make sure the vulnerability assessment tool you pick focuses on Microsoft platforms.

Application-Layer Vulnerability Assessment Tools
Most application-layer vulnerability assessment tools are directed toward Web servers or databases. The difficulty of correctly securing a public Web server can't be overstated. During the past few years, at least a half dozen "security contests" offered would-be intruders cash prizes to attack extremely hardened Web servers. Dream teams of security talent configured the Web servers, which ran hardware protection that most companies can't afford. Yet by my count, in five out of six contests, the system succumbed to dedicated crackers within a few days. Most Web servers fell because of exploits in the underlying OS or holes in the e-commerce application. If the best of the best can't properly secure a Web server, how can the lay Web master secure one? The answer is to run a vulnerability assessment tool built specifically for testing Web servers.

Web-server vulnerability assessment tools are usually targeted to IIS, Apache, or iPlanet. IIS-oriented vulnerability assessment tools will attack poorly configured anonymous user accounts, incorrect directory rights, leftover sample code, and privileged services such as Internet Server API (ISAPI) filters. Apache and iPlanet tools check for chunk code file exploits, attacks against the cgi-bin directory, or directory transversal attacks against /etc/passwd. Vulnerability assessment tools for any Web server will always check for sensitive information stored in hidden fields, stored passwords, cross-site scripting, unchecked inputs, and buffer overflows.

The appearance of SQL injection attacks, and now the Slammer worm, prompted several vendors to produce application-layer tools that specifically test the most popular databases: SQL Server, Microsoft Exchange Server, Oracle, IBM Lotus Domino, Oracle PL/SQL, Sybase, IBM DB2, and MySQL. The tools test for missing or default passwords, injection problems, and poorly configured security.

Password and Account Checkers
A small subset of vulnerability assessment tools will attack your system by guessing passwords. This approach might seem simplistic, but passwords are often the weakest security link on a network. A private key with a weak password can foil even the strongest encryption because the administrator account can't be locked out by a certain number of invalid password attempts, and someone can guess all day long for the password on a directory share without causing any undue notice. Some of the host-based vulnerability assessment tools will also list inactive accounts that are still on the system. Used together, password checkers and account status checkers are a quick way to test and strengthen any network with a minimal amount of effort and money.

Of course, many vulnerability assessment tools cover more than one category. Only rarely will a host-based vulnerability assessment tool not also check for commonly exploited applications on the same host on which it found the OS. Many Windows-specific host-based tools are also password and account-status checkers. But tools that try to do too many tasks sometimes don't do any specific task well. A vulnerability assessment tool built specifically to analyze a Web server or a database server will probably do a more thorough scan against that particular type of host than will a general host scanner, but exceptions exist. No matter what categories a vulnerability assessment tool fits in, it must fulfill the following three roles: map the network and identify the application, test vulnerabilities, and report the findings.

Host-based vulnerability testers begin their work by asking either for the IP address of a specific host to scan or for the subnet range of the hosts to examine. A well-coded, host-based vulnerability assessment tool will find all the physically connected hosts on your network and report on the OS platform type and version (called OS fingerprinting). Vulnerability assessments will usually ping (submit an Internet Control Message Protocol—ICMP—echo) hosts, then start identifying active TCP and UDP ports. Some will automatically assume that a service running on a standard (well-known) port is a particular type of service. For example, a weakly written vulnerability assessment tool will try to attack all services running at port 80 as if it were an HTTP server, even if it is an SMTP server. Some vulnerability assessments will attempt to identify the application running on a particular port (for example, which Instant Messaging—IM—client is active). Ideally (for performance reasons), if the tool can recognize the application and the version, then it will attempt only the attacks that are specific to that application.

Unfortunately, vulnerability assessment tools are poor across the board at identifying applications and ports, and if they can't identify these entities, how are they going to test for the correct vulnerability? The answer is that some vulnerability assessment tools test for all vulnerabilities, even those that don't apply to a particular platform or port. This all-encompassing approach isn't necessarily bad, aside from the associated performance hit. I'd rather a vulnerability assessment tool be accurate than fast.

Choosing a Vulnerability Assessment Tool
Vendors design vulnerability assessment tools to test a host or application for vulnerabilities. Vendors claim their tools can test from dozens to 30,000 different vulnerabilities, depending on the product. Any Windows vulnerability assessment tool should contain several hundred different attack signatures. Although tools that test for each platform's known exploits*as tracked by Microsoft, MITRE's Common Vulnerabilities and Exposures (CVE—a list of standardized names for vulnerabilities), or the CERT Coordination Center (CERT/CC)—have their place, a strong vulnerability assessment tool must go further and look for weaknesses that are attributable to poor configurations. For example, I want to know whether I have blank passwords, misconfigured permissions, open network shares, or an enabled guest account. Some vulnerability assessment tools let you customize the attack scripts, either to fine-tune penetration tests or to avoid false positives.

To be useful, vulnerability assessment tools need to not only report and summarize the found weaknesses of a particular system but also give the administrator help in fixing the holes. Some vulnerability assessment tools I've used report open shares but don't let me know the names of the shares or why they were considered open. Is the tool referring to directory shares or printer shares? Are the shares without passwords, or have I not applied the Microsoft patch to remove share password enumeration? The better tools link to the vendor's Web site, which provides specific details about the exploit, its level of risk, and its remediation. Some tools even design scripts that you can deploy to close the holes across your network.

Picking a vulnerability assessment tool is like picking an enterprise antivirus scanner. Several good vendors exist, but insist on a try-before-you-buy period before committing funds. What works well in one environment might not work well in another. Here are some general comments to consider when shopping for a vulnerability assessment tool.

Pick a tool that matches your environment and expertise. Be sure to find out which platform the vulnerability assessment tool runs on and scans for. As I mentioned, if you manage a pure Windows environment, pick a tool with a Windows bias that runs on Windows machines. Nessus is a popular free vulnerability assessment tool that tests more than 140 Windows exploits. Although Nessus has a Windows monitoring client, you'll need a UNIX or Linux host to run the server portion of the program.

Many of the Windows vulnerability scanners seem stuck in a time warp and are more at home testing Windows NT than Windows XP and Win2K. Make sure your vulnerability assessment scanner is aware of the new OSs and tests their particular nuances (e.g., Universal Plug and Play*UPnP—port 445 NetBIOS traffic, Group Policies). Also, most vulnerability assessment tools work only with the TCP/IP protocol. If you have a Novell or Macintosh network, consider the impact of being able to scan only IP hosts if you run other or additional protocols.

Make sure the vulnerability assessment tool you pick has accurate network and application mapping and penetration tests. How can you discover a tool's functionality? Search the Web, read magazine reviews, and ask the vendor for client references. Test a vulnerability assessment tool's performance before you buy. Most vendors offer a free trial of their fully functional software, limited only by a narrow IP range.

If you plan to scan more than a few dozen hosts at once, make the vendor give you a fully functional product to test. Some tools do a great job of scanning one or a few machines at a time but become sluglike or fail when scanning large subnets. Most enterprise-class vulnerability assessment tools let multiple vulnerability scanning servers be distributed throughout the organization, all reporting to one external database for data collation.

Find out how many vulnerability scripts the tool has for the platforms you're scanning and how often they're updated. Again, pure numbers shouldn't impress you because one vendor might lump many related exploits into one vulnerability test and another might count each test as a different exploit. I like vulnerability assessment tools, such as CVE, that link each test to a standards-based vulnerability case ID. Also, find out how often the product updates vulnerability scans. Find out whether the tool updates scans automatically or whether you need to do this manually. How long after a new exploit such as Slammer comes out will you need to wait to test your entire network? Can you add manual entries and modify existing entries to rule out false positives and fine-tune the scans?

Find out how many reports you get, what information they contain, and whether you can export the reports. Although the canned reports might initially serve your needs, if you frequently test for vulnerabilities, you might want to export data to external databases for further analysis or trending. Most vulnerability assessment tools can't link one scan session to another, but this functionality is essential for most administrators to prove to their CEOs that their discovery and patching rounds are becoming more efficient.

Find out whether the tool provides different levels of penetration to prevent lockups. All vulnerability assessment tools will warn you that the penetration testing process could cause DoS attacks or a system hang on the inspected hosts. Running a vulnerability assessment tool against your mission-critical servers during high usage times isn't a good idea. Vulnerability assessment tools will cause problems and will cause service disruptions and lockups. Most good vulnerability assessment tools let you choose a less invasive penetration test designed to prevent operational disruptions. Consider using the hard-core penetration testing during the initial scan that you run during an off-hours marathon, then use the less invasive scans for subsequent follow-ups.

Find out whether the vulnerability assessment tool's pricing fits your company's needs. Some vendors price vulnerability assessment tools by the number of hosts you plan to scan, and other vendors set pricing by the size of your subnet. Most vendors sell the products through annualized subscriptions. I ran into a licensing problem a few years ago when I placed my Class C—sized network on a private Class A address scheme. The vulnerability assessment vendor at the time had to make a special code fix so that I could deploy his tool without having to buy thousands more licenses than I needed. Whereas most vendors let you run unlimited scans during the licensing period, others charge by the scan. Here are other questions you need to ask about pricing: How much does the vendor charge when you add new hosts? How long does it take to get new licenses? How long are vulnerability assessment updates included for free?

Determine whether you want an agentless tool. Hard-core vulnerability assessment enthusiasts believe an agentless tool is the only way to go. Agentless tools know nothing about the host and will interrogate it just as an intruder would. Agentless tools provide a truer test of a particular machine's exposure. Some vulnerability assessment tools require client-side agents to be installed in order to be highly efficient. Agent-based tools can be more accurate because they can work on both sides of the network interface and discover processes, services, and ports that an outside service couldn't detect. Several vulnerability assessment tools offer both modes.

Consider whether you want the tool as an online service. You can buy a vulnerability assessment tool as an online service. The upside of this approach is that the online tools don't take up hardware, you can run them from any location, you can run reports from any location, updates are automatic, and overall total cost of ownership (TCO) can be lower. On the downside, services tend to scan slower and, in general, aren't as customizable as client-side products. Also, using an online service would put your list of discovered vulnerabilities in the hands of a third party.

A Solid Testing Strategy
At minimum, most network administrators need to conduct annual or quarterly vulnerability assessments against any exposed mission-critical system. You need to thoroughly test high-risk systems, such as Web servers on the Internet or demilitarized zone (DMZ), before deployment and frequently thereafter. The vulnerability assessment tool's purpose is to notice the exploit hole before a malicious intruder does. Your vulnerability assessment strategy needs to go beyond one initial test and fix-it round. If you conduct your security audits correctly, subsequent audits will go quickly because the old holes will be closed and the vulnerability assessment tools will have less work to do.

Customers are demanding more from their vulnerability assessment tools. First, customers increasingly expect vulnerability assessment tools to help automatically fix security holes they find. Vendors are providing proactive solutions in different ways. Some products point the network administrator to the correct patch to close the hole. Others are going as far as to write scripts to configure weak workstations or scripts that will interact with other security devices, such as firewalls and Intrusion Detection Systems (IDSs). In addition, vendors are responding to customer demand to integrate vulnerability assessment tools into enterprisewide monitoring systems with centralized report, alerting, and management. A large management system documents the company's security policies, and the vulnerability assessment tool ensures those policies are being applied. The centralized security console monitors the vulnerability assessment tool, the firewall, the IDS, and the antivirus scanner, and coordinates responses from all. When a vulnerability is found, the assessment tool is being asked to make the risk decision for the company. Is an unpatched SNMP agent a serious threat in your environment? Soon, your company might expect the vulnerability assessment tool to be the expert.

Vulnerability assessment tools are crucial for testing and maintaining strong network security. Vulnerability assessment tools explore the network, map available hosts, and exploit found systems and software. Table 1 lists some products to check out if you're looking for a vulnerability assessment tool for your company. After you find a tool that supports the platforms and applications you'd like to test, give it a try by setting up test machines across your network with known vulnerabilities. No vulnerability assessment tool is perfect, so pick one that best matches your company's environment, budget, functionality, and accuracy and reporting needs.