Take Advantage of the EventCombMT Utility

If someone asked me to name the administrative tasks that I despise, I would to have to include searching event logs. Using the log-search tools included with most Windows products to find a specific event-log entry requires a combination of skill, guesswork, and luck—and the larger the environment, the more difficult the search. Large organizations tend to solve this problem by using expensive log-consolidation servers, databases, and search tools—a solution that rivals many modern business intelligence (BI) systems and their data-mining efforts. Smaller organizations with limited resources tend to depend on the built-in Event Viewer tools and simple log utilities.

Searching event logs doesn't have to be complicated. The Microsoft TechNet Web site offers a simple, effective utility called EventCombMT as well as event-log searching tips. Best of all, the utility and tips are free. They're part of the Microsoft online guide "Security Operations Guide for Windows 2000 Server" (http://www.microsoft.com/technet/security/prodtech/windows/windows2000/staysecure/default.asp). After you go to this Web page, you can download the .zip file (secops.exe) that contains EventCombMT by clicking Download the associated Scripts. (If you want to access secops.exe directly, go to http://www.microsoft.com/downloads/release.asp?releaseid=36834.) Secops.exe contains several utilities; you'll find EventCombMT in the securityopseventcomb folder. You can use this handy utility's many options to perform built-in and custom searches.

Using EventCombMT's Options
EventCombMT lets you query event logs from multiple computers (servers and workstations) for a specific event ID, a range of event IDs, or specific text within an event. After you launch the utility by double-clicking eventcombmt.exe, it will query you for your current domain and prepopulate a computer list with known domain controllers (DCs). If you need to use a different set of credentials, such as your Administrator account, you can specify those credentials by selecting the Use Alternate Credentials option on the Options menu.

To use EventCombMT, you first need to select the appropriate computers to search. Although the utility prepopulates the computer list with DCs in your domain, you can search virtually any DC, member server, or workstation for which you have credentials. The Win2K default settings require that you be an administrator or domain administrator to query a local computer's Security log or a DC's Security log, respectively.

To add a machine to the computer list, right-click inside the Select To Search/Right Click To Add box. As Figure 1 shows, this box is just below the Domain text box. Select Add Single Server. In the Add Server dialog box, enter the name of the computer you want to add in the Server Name text box. Although the text box is labeled Server Name, you can enter the name of any DC, member server, or workstation running Windows NT 4.0 or later. (If you don't know the computer's name, you can click Browse and navigate to the computer.) Click Add Server, then click Close. The Select To Search/Right Click To Add box should now include that computer's name. Select that computer by clicking it.

Next, you need to select the log files for which you want to search on the computer you just selected. If you selected a workstation running Windows XP or Win2K Professional, or a computer running NT Workstation 4.0 or NT Server 4.0, you can search only the System, Application, and Security logs. If you selected a DC running Windows .NET Server (Win.NET Server) 2003 or Win2K, you can search three additional logs: FRS, DNS, and AD. Searching the FRS log lets you monitor the health of the File Replication Service (FRS), whereas searching the DNS log lets you monitor DNS replication traffic. If you want to find duplicate SIDs in Active Directory (AD), you can search the AD log.

After you select the log files, you'll likely want to concentrate your search by specifying the types of events for which you want to search. Audit events are typically recorded as either a Success Audit or Failure Audit. For EventCombMT to return audit events, auditing must be enabled on the target computer. Auditing is enabled on a per-computer basis unless Group Policy Objects (GPOs) override the local audit policy. Error, Informational, and Warning events are usually related to system events.

Next, you can narrow your search by filling in the Event IDs, Source, Text, and Scan Back options. Entering the appropriate event ID in the Event IDs text box is the most important way to narrow a search. You can enter one event ID or multiple event IDs. (If you list multiple event IDs, separate them with spaces.) To search a range of event IDs, you can use the > ID < option.

The Source option lets you select a specific process (e.g., Print, SAM, Schedule) to search. By default, EventCombMT searches ALL SOURCES, but you can change that default by selecting a process from the drop-down list. If you want to search for a specific string in a log, you can enter that string in the Text string box. You can also specify how far back you'd like to search through event logs. After you decide how far back you'd like to search, enter that information in the Scan Back text box. Be sure to select the appropriate Minutes, Hours, or Days option.

If you aren't quite sure what event you're looking for, you can select the Get All Events With Above Criteria check box to prompt EventCombMT to download the entire event log. You'll notice that selecting this check box disables the Event IDs, Source, Text, and Scan Back options.

You can control the number of concurrent threads dedicated to a search by using the Threads slider. The default is 25 threads. You can change that number before or during a search. For example, if you find that a search is too heavily taxing your system, you can decrease the number of dedicated threads during the search. Doing so won't kill any running threads; it only prevents additional threads from starting until the usage drops below your new limit.

By default, EventCombMT outputs search results to a comma-delimited text file in the C:temp directory. You can change the output directory by selecting the Set Output Directory option on the File menu. If you're running multiple searches and need to keep the output separate, you can take advantage of this feature and use separate directories for the searches.

Now that you know about EventCombMT's options, let's look at an example of how to use it. Suppose you want to look for user allenj's logon events for the past month on computer AllenJ1. Select that computer, then select the Security log to view the list of users logging on to that computer. In the Event Type section, select all the check boxes, as Figure 1 shows.

Next, make sure that the Get All Events With Above Criteria check box is clear. Then, in the Event IDs text box, enter 528 (for computers running NT 4.0) or 540 (for DCs running Win2K or later) to search for successful logons. In the Text text box, enter allenj. Enter 30 in the Scan Back text box, and select the Days option.

Now you're ready to kick off the search. After you've double-checked your entries, click Search. You'll see a lot of activity both on the right side of the EventCombMT dialog box and in the Status field. If you want to know more about what's going on in these areas, see the online Help documentation.

After the search is complete, the utility automatically opens the output directory in Windows Explorer, so you can quickly see the results. EventCombMT outputs each computer's event log to a separate .txt file. If separate files are a problem, you can redirect the output to one file after you've finished running EventCombMT. At the command line, enter a command such as

type source-text.txt >> destination.txt

where source-text.txt is the EventCombMT output file whose output you want to append to the new output file destination.txt. You need to run this command for each EventCombMT output file that you want to add to destination.txt.

Using the Built-In Searches
EventCombMT contains built-in searches, such as the search for duplicate SIDs in AD, FRS failures, and hardware disk errors. One of my favorite built-in searches is the search for account lockouts. Finding which DC is responsible for locking out a user and the events leading up to the lockout isn't an easy task if done manually. To load this predefined search, select Built In Searches, Account Lockouts on the Searches menu. The utility automatically selects the DCs and the account-lockout events that Table 1 lists.

Accounts typically get locked out in the following manner: When a user attempts to log on and fails because of a bad password, each attempt is logged with event ID 529. After the account has exceeded the maximum number of attempts and the account is locked out, the PDC emulator records event ID 644 to mark the occasion. Subsequent attempts to log on using the locked out account are logged on the DC with event ID 539.

Event IDs 529 and 539 are among the most difficult to find because they're logged locally at the computer at which the logon attempt occurs and might not even be logged at all if the account lockout occurs on a computer that doesn't audit logon and logoff events. However, you can always locate event ID 644 entries on the PDC emulator. The event ID 644 entry should tell you on which machine to look next. If that machine isn't auditing logon and logoff events, you need to enable the logging of these events to investigate future account-lockout events.

In large network environments, searching for individual account failures can be quite time- and resource-consuming. When investigating locked-out accounts in such an environment, start by searching for event ID 644 first to learn details of the lockout, including the machines that tried to perform the authentication. Then, expand your search to event ID 529 to see the individual failed attempts. Other events can also indicate a failure to authenticate, as Table 2, page 16, shows. The events in Table 2 are logged locally at the computer at which the logon attempt occurred.

As I described earlier, event IDs 528 and 540 denote successful logons. Event ID 528 is logged locally at the computer at which the logon occurred, whereas event ID 540 is logged at the appropriate authenticating DC. Within each event, the log records the type of logon. For example, Web Table 1 (http://www.secadministrator.com, InstantDoc ID 37450) lists the logon types for event ID 528. You can use these event IDs and logon types to build an account history for a user.

Building Custom Searches
In addition to using EventCombMT's built-in searches, you can build and save custom searches. For example, if you want to learn when new accounts are created, you can search for event IDs 624 and 626. You can learn when users change their passwords by searching for event IDs 627 and 628.

Shutdowns and blue screens are worth searching for and investigating to find possible service problems. Table 3 shows the event IDs for these events. Finding event ID 6008 without a corresponding event ID 1001 might represent a hung server, or the server might have been powered down.

Occasionally, shutdown and blue-screen events can be evidence of a security problem. A malicious user might be trying to boot to another OS or otherwise interfere with the typical startup sequence. For security purposes, you'll also want to monitor event logs for event IDs 612 and 517. These events represent a change to the audit policy and a clearing of the Security log, respectively. Malicious users and intruders will often try to change what's being audited to minimize what's logged or clear the event log to cover their tracks. If a log has been cleared, the log will include not only event ID 517 but also the user account that was used to clear the log.

Be Proactive
Frequently checking your logs for failures and suspicious events exemplifies good administration skills and helps you proactively monitor system security. Using EventCombMT to search many computers at once can quickly give you a better picture of what's going on in your environment, enabling you to react faster to incidents.