Collecting and Analyzing Event and System Logs

Managing multiple servers has never been easy for businesses of any size. The problem is perhaps most acute when it comes to managing audit and event logs. These logs, especially on Windows systems, contain a wealth of information valuable to administrators. Information of interest ranges from details of successful and unsuccessful logon attempts; to use of specific privileges; to creation, management, and deletion of users and groups; to group membership changes.

Although it's easy to configure what's logged on Windows systems, it isn’t as easy to collect that information in one place and mine it. Consequently, large enterprises often end up spending big sums of money on complex solutions. Small and midsized businesses (SMBs) can take advantage of free tools distributed by Microsoft to make the task easier. These tools can also be useful in large enterprises when ad hoc event log collection and mining is required.

In this article, I’ll introduce you to the basics of the Windows event log system; introduce you to EventCombMT, a free tool from Microsoft that can make the job of collecting and mining event log information easier; and offer some suggestions for how to manage and configure event logging and collection across several systems from a central location.

Introduction to the Windows Event Log System
Understanding how the Windows event log system works will help you make decisions about how best to manage it, including evaluating tools and products from Microsoft and third parties designed for this task. The event log is actually an elegant and extensible system that lets Microsoft and third-party applications integrate their error reporting with Windows' error reporting. On Windows 2000 and later, you can view the event log by running the Event Viewer (eventvwr.msc) from the command line or by using the Microsoft Management Console (MMC) Computer Management snap-in. A standard Windows XP system has three logs: Application, Security, and System. Applications can create additional logs. A Windows Server 2003 system has the same three logs, unless it's a domain controller (DC), in which case, you’ll also find Directory Services and File Replication Services logs. If you install DNS on a Windows 2003 system, you’ll also find a log called DNS Server.

The Windows OS and applications that leverage the event log install message files. Each message file, typically a .dll or .exe file, contains a string table that maps event categories to category names and event identifiers to error messages. During the installation process, applications register their message files with the event log service in the registry under the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesEventlog subkey. Under this subkey, there is a subkey for each event log, and applications create a subkey under the appropriate event log subkey. Applications place their event log registration information under their specific subkey.

When Windows or an application wants to create an event log entry, it opens the event log it wants to write to and writes an event specifying the event source, event category, event identifier, strings specific to the event itself such as filename or username, and any additional raw data that's relevant. When Event Viewer or another application reads back event log entries, it must use the event source information contained in the entry to query the registry for the location of the message files specific to the application that generated the event, load the event message from the message file by using the event identifier as a key, and format the output by using the event message from the message file and strings specific to the event.

This mapping between application sources and message files stored in the registry and the need to load event messages from message files to properly format an event log entry for presentation to the user are what make event log consolidation so problematic. Consider this: If you archive an event log to an .evt file and copy the file to another system, you'll encounter errors when reading events in Event Viewer if you don’t have a local copy of the message files and the appropriate registry entries for the source application. Figure 1 shows such an error. To consolidate event logs and mine the data in them, you must use the application-specific message files installed on the source systems and retrieve the formatted event entries. When you view an event log remotely by using the Event Viewer, it's smart enough to open the remote computer’s registry, locate the message files, and open them remotely. If Event Viewer can't perform one of these actions, you'll see an error similar to the one in Figure 1.

Collecting Event Logs from Remote Systems with EventCombMT
Although you can use Event Viewer to open remote event logs, it's not suitable for mining for information in a timely fashion when a large number of systems are involved—for example, when investigating a security breach in which a hacker might have logged on to multiple systems. A better tool for this is Microsoft's EventCombMT, which is available in several downloadable toolsets, including the Windows Server 2003 Resource Kit Tools. The most functional version of EventCombMT is the one that ships with the Account Lockout Tools (ALTools.exe), which are available from http://www.microsoft.com/downloads/details.aspx?displaylang=en&familyid=7af2e69c-91f3-4e63-8629-b999adde0b9e, and this is the version that I recommend you download and use.

EventCombMT, shown in Figure 2, is a multithreaded GUI utility that was originally designed to run predefined common queries against event logs for events such as account lockouts and File Replication Service (FRS) errors. It can process event logs from up to 100 remote computers at a time. Despite being designed with predefined queries, EventCombMT is remarkably flexible and can be configured to pull event logs from remote systems determined by a complex set of criteria.

For example, if you're interested in all security events from all event logs across all the systems in your domain, first enter the domain name in the Domain field, right-click in the Select To Search/Right Click To Add field, then select Get All Servers (Slow) from the context menu (which Figure 2 doesn't show). This method will query the domain Master Browser for all servers in the specified domain. This isn't the most reliable method of identifying all servers in your domain, so you might want to manually compile a list of servers, save it to a file, then select Get Servers from File from the Select To Search/Right Click To Add context menu instead. You can also manually add servers by selecting Add Single Server from the context menu, then entering servers one at a time.

Next, select the Security option under Choose Log Files to search, the Success Audit option and Failure Audit option under Event Types, and the Get All Events With Above Criteria option. When you select this option, the Event IDs, Source, and Text fields are disabled. If you want to narrow your security event search, you can leave the Get All Events With Above Criteria option clear and specify event IDs of interest (security events in the Windows OSs begin with event ID 512 and continue to the high 600s) and the source (most likely Security). You can list event IDs individually (separated by commas) or specify a range. Once you've entered your search criteria, click Search. If you haven’t selected servers, an error message dialog box will ask if you want to select all systems.

As EventCombMT runs, it generates statistics that you can view. EventCombMT runs in the context of the user who launches it. It's possible to specify credentials for another account by selecting Use Alternate Credentials from the Option menu. You can store credentials in the registry, but I recommend against it because stored credentials aren't adequately secured.

Events that match the specified criteria are written to a text file. If you specified Security under Choose Log Files to search, the output file will be named hostname-Security_LOG.txt. If you select other criteria, the filename will change to represent the criteria you select. The text files are actually in comma-separated value (CSV) format. If you prefer, you can specify that files are written with a .csv extension by selecting Save Files as CSV Files from the Options menu. The output files are written to the folder C:temp by default, and this folder is created if it doesn’t exist. You can change the output folder by selecting Set Output Directory from the Options menu.

EventCombMT can also store the event log records it retrieves in a Microsoft Access database—if you select Write All Results to Database from the Options menu. EventCombMT creates a database named EventCombMT.mdb. The database contains a single table called EventCombMT with the columns Server, LogFile, EventID, RecordType, EventSource, EventDate, User, and EventText. If the EventCombMT.mdb database already exists, EventCombMT will ask you if you want to overwrite the database or add the records retrieved to it.

You can also specify dates between which you want to fetch events by selecting Set Date Range from the Options menu. You can set a date range only if you clear the Get All Events With Above Criteria option. This means you'll also need to specify the event IDs you're interested in.

Once you've tested EventCombMT and found the search criteria that suits your needs, you can save the search by selecting Save This Search from the Searches menu. You'll be prompted to provide a name for the search. Saved searches are per-user and are stored in the registry under HKEY_CURRENT_USERSoftwareMicrosoftEventCombMT. You can use a saved search to search only the servers specified in that search's criteria—you can't use the search on a different group of servers. You can load a previously saved search by selecting Load A Search from the Searches menu and selecting the desired search from the list of saved searches. An EventCombMT quirk stops the tool from storing events in a database when working with saved searches unless you also select Decode Event 1000 Flags from the Options menu.

Scripting and Scheduling EventCombMT
As useful as the EventCombMT GUI is, you're not likely to want to manually run it on a daily or weekly basis to collect events from event logs. Fortunately, EventCombMT has command-line options that permit you to script it and use the AT scheduler or Scheduled Tasks to run the script on a regular basis. The simplest means to launch EventCombMT from the command line is to run

EventCombMT  /load:”saved search name”  /start

This tells EventCombMT to run the saved search immediately. If you run EventCombMT in this manner from Scheduled Tasks, remember to specify the credentials of an account that has the search saved in its profile. You won't be able to use the AT scheduler to run an EventCombMT saved search because jobs launched using the AT scheduler run in the context of LocalSystem. Also be aware that if you plan to use Scheduled Tasks to run saved searches that store results in a database, and if the database already exists, EventCombMT will prompt you to extend the existing database, delete and recreate it, or cancel the operation. Because Scheduled Tasks aren't interactive, the EventCombMT task will hang indefinitely waiting for a response. Thus, you must ensure that the existing database is deleted, renamed, or moved before the Scheduled Tasks EventCombMT task runs.

In instances where it isn't feasible to specify a saved search when scripting EventCombMT, you can specify most options from the command line. The first thing you'll want to consider is how to specify which systems to run EventCombMT against. You can use the /DC option to cause EventCombMT to run against all domain controllers (DCs) in the domain in which EventCombMT is running. You can specify an alternative domain by using /DC:domainname. If you want to run EventCombMT against a handful of servers, you can use the /S:servername option for each server. If you have a lot of servers to specify, you can use the /FILE:filename option. The file specified should contain a list of servers, one per line.

Once you've specified the servers you want EventCombMT to run against, you need to specify the event log files to process. This is done with the /LOG:logfile(s) option. The supported log files are sys for System, app for Application, sec for Security, ds for Directory Services, frs for File Replication Services, and dns for Domain Name System. You can specify multiple logs on the /LOG: option—type the names one after the other with no separation between them. For example, /LOG:sysappsec will cause EventCombMT to process the System, Application, and Security event logs on target servers. If you want to process all the event logs, you can use /LOG:all.

Next, you need to specify the event types you're interested in by using the /ET:eventtype(s) option. The supported event types are w for Warning, e for Error, I for Informational, sa for Success Audit, fa for Failure Audit, and su for Success. You can combine multiple event types or use /ET:all to process all event types.

Last, you need to specify the particular event or events that you're interested in. Use the /EVT:”eventid [eventid] [eventid]” option to list these; separate events with a space.

You can specify additional information such as the output folder by using the /OUTDIR:”path to folder” option, the event source by using the /SOURCE:sourcename option, text to search for by using the /TEXT:”text strings” option, and a date range with the /AFTER: and /BEFORE: options. When specifying a date range, the date should be in the format MMDDYYYYHHMMSS, must be 14 characters in length, and must specify the hours according to the 24-hour clock.

Best Practices When Processing Event Logs
EventCombMT is a great tool for consolidating events from event logs in a central location or in one database. However, you're still faced with the daunting task of going through the retrieved events looking for items of interest. I recommend several practices that will make your job easier.

First, make sure that each system you're pulling events from is configured to log events of interest to you. There's nothing more frustrating than looking through logs for an event that was never recorded in the first place. Microsoft has changed the default logging options for the most recent versions of Windows, so if you run a mixture of Windows 2003, XP, and Windows 2000, you might be surprised that different events are recorded out of the box on each. In small domain environments, the simplest means to ensure that all systems are recording the same events to the event logs is to set the Default Domain Policy, which Figure 3 shows. In larger environments with a mix of servers and workstations, each running unique applications and services, you'll likely need to collect similar servers into organizational units (OUs) and apply a Group Policy Object (GPO) for Audit Policy to each OU.

Second, configure EventCombMT to retrieve events only from servers of interest—not all servers. Cutting the number of servers helps EventCombMT finish processing more quickly and pull in less data, making it easier to mine that data.

Third, configure EventCombMT to retrieve only events that are truly of interest. For example, if you suspect that a user account has been compromised and is being used to explore the network for sensitive data, you need to know which servers the account has successfully logged on to. Thus, you need to look for successful account logon events only—not for failure logon events. You can specify that EventCombMT processes only Success Audit event types for event IDs specific to account logons—not logoffs, account management, or use of privileges.

Fourth, whenever possible, have EventCombMT store all retrieved events in an Access database. You can view and search a CSV file loaded into Microsoft Excel much like you can an Access database, but by having all events stored in one database, you're less likely to miss an event than if you have tens or hundreds of CSV files to process manually and you forget to process one file. Another advantage of having EventCombMT store events in an Access database is that you can use many third-party tools to mine that data.

Last, once you've processed events in CSV files or an Access database, delete, rename, or move the files or database. If you have large files, lots of files, or a large database, specifying a new output folder with /OUTDIR: each time you run EventCombMT might be easier for you.

Further Information
For a more detailed look at using EventCombMT, see the Windows IT Security article "Take Advantage of the EventCombMT Utility," January 2003, InstantDoc ID 37450. The Microsoft Knowledge Base, at http://support.microsoft.com, contains information about how to use EventCombMT to troubleshoot a number of scenarios, including querying account lockout information. There you'll also find information about the event IDs used in the various versions of Windows to record security events: "Windows 2000 Security Event Descriptions (Part 1 of 2)" at http://support.microsoft.com/?kbid=299475, "Windows 2000 Security Event Descriptions (Part 2 of 2)" at http://support.microsoft.com/?kbid=301677, and "Security Event Descriptions" at http://support.microsoft.com/?kbid=174074. The Help file that ships with EventCombMT is small but does include some useful information.