Vulnerability Scanner Showdown: AppDetectivePro 5.4.6 vs. AuditPro Enterprise 4.0

Database vulnerability scanners are tools you can use to assess the configuration of your database servers for security weaknesses that expose them to known threats. These scanners can be a critical tool in avoiding the financial and legal costs companies of all sizes could sustain from a single incident of compromised data. Here, I review two products: Application Security's AppDetectivePro (ADP) 5.4.6 and AuditPro Enterprise (APE) 4.0 from Network Intelligence (India). These products install with predefined tests and sets of tests called policies, which can be part of an ongoing program to minimize the risk of your systems and data being compromised. As you'll see, there are some major differences between these two products. Microsoft's Baseline Security Analyzer v2.1 is another choice for SQL Server security, but both of the products reviewed here include more comprehensive sets of assessment tests for SQL Server.

AppDetectivePro 5.4.6

ADP is a vulnerability-assessment tool that probes your application servers from another system on the network. It tests SQL Server 2005, 2000, and 7.0, including x64, Microsoft Data Engine, and Express editions. The company plans to implement support for SQL Server 2008 in second quarter 2009. ADP also tests several other platforms, including IBM DB2, Lotus Domino, Oracle databases, Sybase ASE, and MySQL. Application Security licenses ADP by the database instance.

ADP installs on systems running Windows Server 2003, Windows XP, or Windows 2000, including x64 editions. By default, ADP stores results in a Microsoft Access database that's created when the product is installed. If you're planning to monitor a large number of databases, you can instead configure a SQL Server database for better scanning performance.

ADP uses a framework of jobs and tasks to perform vulnerability assessments. Session tasks define the applications and IP ports you want ADP to test. Discovery tasks locate systems with active ports on the network. Policy tasks define the set of tests ADP will perform, which can include both predefined and user-written tests. Audit and penetration (aka pen) tests probe the target systems for vulnerabilities: Audit tests assess from within the system, and pen tests simulate malicious hacker attacks from outside the system.

ADP includes a job scheduler that lets you automate audit and pen test runs for ongoing assessment. It also includes a vulnerability manager to track the status of ADP's findings. ADP will generate scripts to correct common configuration problems.

When installed, ADP launches the Discovery Wizard, which prompts you for host names, IP addresses, and database systems to scan. The result of running the Discovery Wizard is a named session that includes each server instance ADP found during the scan. I created a session that included one Win2K system hosting SQL Server 2000 and one Windows 2003 system hosting SQL Server 2005.

The ADP GUI is your primary interface for configuring sessions, running audits and pen tests, and managing reports. As you can see in the screen below, the top of the GUI provides access to a set of menus. A hierarchical view of the currently loaded session is displayed on the left, and a details pane is to its right. By default, ADP places all systems in the session in a folder called Network. You can create a multilevel hierarchy of folders to organize the IP addresses in a session. When you've highlighted a test in the session pane, the Details tab at the bottom of the right pane shows which policy defined the test, a brief summary of testing, and other information. The bottom pane lists the vulnerabilities discovered by the test. Clicking a vulnerability displays its description and suggests remedial action in the details pane's Vulnerability Description tab. You can find similar information in the Vulnerability Details report, which I discuss later.

Once you've created and loaded a session, you can run an audit or pen test using a custom policy or one of ADP's built-in policies. I ran several of each test, most of which took just a few minutes. A test against an old SQL Server 2000 system reported 16 high-risk, 10 medium-risk, 37 low-risk and 10 informational vulnerabilities. Tests against a SQL Server 2005 system reported only three high-risk vulnerabilities (two weak passwords and missing updates) and one low-risk vulnerability (failure to use Windows-only authentication mode). Pen tests run without supplied authentication and can, depending on the particular tests you include in the policy, attempt to crack security and gain access to the system. When you run a pen test, ADP warns you that the testing activity may be logged on the target system.

ADP supports an extensive list of pen and audit tests. Pen tests available when you create a custom pen test policy include Denial of Service attacks, attacks for common user IDs and easily guessed passwords, system configuration problems, and known vulnerabilities. The application divides audit tests into access control, application integrity, identification and password control, and OS integrity categories.

The reporting, based on Crystal Reports and accessed via a test's right-click menu, is thorough and easy to use. After ADP runs an audit or pen test, the results are available for viewing on screen. You can export reports in 16 formats, including PDF, HTML, XML, Excel, and comma-separated value files. After running an audit or pen test, ADP presents four report options. A vulnerability summary is a brief description of the test and a count of vulnerabilities found. A vulnerability detail report lists much more information. For example, the vulnerability detail report for the test permission on registry extended proc listed two extended stored procedures with Execute permissions granted to the Public role, a description of the vulnerability, and sample syntax for aREVOKE T-SQL statement to remove Execute permissions from the procedures. A check-status report includes a two-line summary of each test in the policy with results. A user information report lists all accounts and the passwords the audit or pen test discovered. The report-viewing window has a typical table of contents panel along the right that, by default, displays only the highest level sections preceded by plus (+) signs. I wished for "expand all" and continuous scrolling options here but didn't find them.

The previously described reports all relate to a single audit or pen test run, but the Reports button at the top of the GUI accesses a set of nine reports that are based on data from multiple tests. One useful report, Vulnerability Differences, compares two audits or pen tests done at different times and reports the vulnerabilities that are resolved, unresolved, and new between the two. Another report gives the details of a specified policy. A summary report graphs the vulnerabilities detected during the testing for a session.

ADP has several other features that support your efforts to maintain secure systems. The Vulnerability Manager feature lets you focus on a particular area of interest by filtering test results by subjects such as IP address, vulnerability, or severity. The Fix Scripts feature generates scripts that you can customize and manually apply to correct common configuration problems. You can also create custom tests that consist of a SQL query and a set of criteria to apply to the result set. Another feature facilitates download and installation of program updates, ensuring that you have the most current program code and testing rules.

I am impressed by ADP. It's easy to install and very easy to use. I found the PDF Help documentation and the GUI's Help files useful. I appreciated that ADP can assess a system by using authenticated access from the inside as well as attack the system using unauthenticated access from the outside. ADP includes a useful set of reports and the flexibility to export them in your favorite format. I think many users will appreciate that ADP not only provides a full description of reported vulnerabilities but also suggests remedial action, including providing SQL statement syntax where appropriate. This application is easy to recommend. When you're looking for a database vulnerability scanner, let AppDetectivePro be the first one you evaluate.

AuditPro Enterprise 4.0

Network Intelligence (India) describes AuditPro Enterprise as a security audit tool rather than a vulnerability assessment tool because APE uses administrative credentials to authenticate to the system being tested. Testing in this way lets the product thoroughly assess the system's configuration, including patch levels, registry contents, and NTFS and database permissions. However, APE doesn't simulate attacks from outside the system.

APE assesses systems running under Windows 2003, XP, Win2K, Red Hat Linux, and Solaris. It assesses SQL Server 2005 and 2000, Oracle databases, DB2, and Cisco Systems routers. The application includes a vulnerability database, which it uses to check for known conditions. Options in the GUI let you check for updates at startup or on demand.

In APE, named policies define the tests that will run against a particular system type out of those that APE can assess. Selecting one of the options displays the sets of "probes"—individual tests—appropriate to that system. You complete a policy by selecting the probes you want to run. APE includes 74 probes for SQL Server systems—a goodly number—and a separate set of probes for Windows OSs. OS probe sets also include network-related testing for open ports. APE targets ISO/IEC 27001 compliance and includes probes in each of the key application areas in support of that compliance.

Named profiles define the systems you want to test, together with the needed authentication credentials and the policy defining the probes you want to run against the system. APE lets you add individual systems by NetBIOS host name or IP address and choose from a display of host names for a domain from Active Directory (AD). I was surprised to discover that APE doesn't let you enter a DNS-style host name. When a system is running more than one supported application that you want to test within a profile, such as Windows and SQL Server or SQL Server and Oracle, APE lets you provide credentials and select a testing policy for each application on the system. Your completed profile is a list of systems, each with one or more applications, along with necessary authentication credentials and testing policy.

APE seems inconsistent in some features, such as its support for Sybase. For example, the application shows Sybase as an option in Profile Manager but not Policy Manager. The AuditPro Enterprise GUI, shown in the screen below, could be more user-friendly. For example, the main UI, Policy Manager, and Profile Manager are implemented in windows that you can't resize, and several columns are too narrow to show all the information.

The documentation for APE is inadequate. Documentation available on the Network Intelligence (India) website describes the product but provides no usage information. When I requested more documentation, the vendor sent me an eight-page PDF that described the installation and usage cycle for APE in general terms. The Help installed with APE provides information about probes for Windows, Linux, and DB2, but not for SQL Server or Oracle databases or for using Probe Manager to create custom probes.

I installed APE on a Windows Server 2003 system running SQL Server 2005. I had to run two installation modules, one for AuditPro, and a second one that the vendor described as a Crystal Reports module for advanced reporting features. I used the AD discovery feature to create a profile that included eight Windows systems with a minimal set of probes for each, a profile with a Win2K Server system with SQL Server 2000, and a third profile with a Windows 2003 and SQL Server 2005 system. The latter two I configured with the full set of probes for their versions of Windows and SQL Server. After running the audit for each profile, APE generated a set of HTML reports. The high-level report reported the number of vulnerabilities detected in five categories, with a link to a detail report for each system and application—in this case, one report for the Windows probes and a separate report for the SQL Server probes. In each case, the tests for a single target system completed in only a few minutes, and APE reported a set of vulnerabilities similar to that reported by ADP. The summary results matrices in both the GUI, shown in the screen above, and the summary report include hotlinks to the detail information, but the links produced only blank reports.

The detail report lists the detailed results of each probe, reporting and assigning a severity level to each vulnerability tested and giving a value of OK or a risk level of low, medium, or high. APE also calculates a weighted vulnerability score, assigning a value of 1, 2, or 3 to low-, medium-, and high-risk vulnerabilities, respectively, and reporting the total. For Windows, the report lists security-related updates, open IP ports, running services and processes, local security policy settings, administrative users, and an analysis of event log settings, to name a few. For SQL Server, the report includes authentication and authorization information, such as members of the sysadmin role, execute permissions assigned to various stored procedures, the state of login auditing, and SQL Server trace settings.

I eventually discovered a Settings menu option, used to change the console password and create a database where APE stores audit results, a procedure described in updated documentation I eventually received for APE. I created the database, which included four user tables. After I ran some audits, the Advanced Reporting option presented the audits as available for comparison and reporting, but none of the four report types (generic report, trending analysis, audit results report, software inventory) produced a report. Considering the absolute dearth of documentation around the installation, configuration, and use of the reporting component, I chose not to investigate further.

Overall, APE isn't quite ready for prime time. Perhaps if the documentation were complete enough to describe how to configure and use the product in greater detail, I'd have had a different experience. As it is, the HTML-based detail reports generated directly by running an audit provide a lot of useful detail information and can be used as a checklist to lock down security on a system.

A Clear Winner

APE can't do penetration tests but generally performs well as a scanner. It can detect a wide variety of vulnerabilities and helps with ISO/IEC 27001 compliance. However, APE's interface is difficult to use, its documentation is incomplete and insufficient, and some of its reporting features seem nonfunctional. ADP surpasses APE with audit scans that are at least as good as APE's, extensive penetration tests, a much better UI, and excellent documentation. While APE generally tells you what to do to fix problems, ADP provides more thorough explanations and, frequently, step-by-step procedures. I have to award my Editor's Choice to ADP. In all respects, it's the more complete product.