Security Log Collection

The growing requirement for statutory and regulatory compliance has many systems administrators scrambling for a way to manage their security logs more effectively. Security logs can be a great way to prove compliance and detect noncompliance. Even for companies that aren't subject to regulatory compliance requirements, security logs are an effective means of detecting violations of security policy and attempted and successful intrusions by attackers. The Windows event logs provide comprehensive security logging, but managing logs across many systems has always been challenging.

Solutions have sprung up to address this difficulty. Many are designed for larger enterprises and are quite expensive; fewer are what I would consider affordable for small-to-midsized businesses (SMBs). Fortunately, some free tools fill the gaps.

Let's look at how to identify some criteria you can use to help choose a security log management tool. I'll also show you some free and low-cost security log management tools and describe some security log management best practices.

Evaluating Your Security Log Management Needs
Before you begin testing security log management tools, you should spend some time thinking about your security log data collection requirements. The following five steps outline a process that can help you identify which security event data you need to collect. This exercise will give you some criteria to use to determine which collection tool is right for you.

Step 1: Identify the security event data you're interested in. Windows and other systems generate a wealth of security data. Typically you'll want to log the results of authentication and authorization operations, such as user logon and logoff activity; access to restricted pages on Web sites; and database and Internet access. Think about what data you actually need. Do you need all authentication and authorization-related data, or are you interested in only failed attempts? Do you need both logon and logoff data for all logon and logoff events (e.g., interactive, network, services, batch) or only interactive logon data?

In this step, I recommend thinking at a high level instead of very specifically—for example, think failed user authentication instead of failed interactive logon to a Windows XP Professional Edition desktop—so that you'll be less likely to miss or overlook sources of data. If you aren't sure what security event data you actually need, consult business decision makers and other stakeholders, such as your inhouse legal counsel or audit group for guidance about what data might be required to prove compliance or look for policy violations.

Step 2: Identify available security logs and the format of each. You might find that there are more security logs in your environment than you originally thought. For starters, there's a Security event log on every Windows system. On UNIX and Linux systems, there's probably a file called Messages (or something similar) that might contain security information. Microsoft IIS and Internet Authentication Service (IAS) also have log files containing security information. These logs can be written in a variety of text and database-compatible formats. You can use the Microsoft Management Console (MMC) snap-ins for IIS and IAS to configure the format used by the services. Generally, it's easier to search text files for items of interest.

Firewalls, access points (APs), routers, and so forth typically create security log data. If these are devices that send their security log data to a logging host—as opposed to services running on a host—note the protocols used and make sure that your log management system can support those protocols. Chances are the devices use the syslog protocol (or a variant of it) or SNMP. You should also record potential sources of security log data, even if the data isn't currently available. For example, a Web server or database might be able to generate security log data even though it isn't currently configured to do so.

Step 3: Map the needed security data to its specific sources. If you need to manage security log data to identify potential attacker activity related to failed authentication attempts, list each source that provides that data. You also might be able to eliminate some sources of data in this step. For example, if you decide to keep a record of user logon and logoff activity in your forest or domain, you could record all activity from the Windows Security event log on each XP and Windows Server system. Alternatively, if everyone has a domain account, each system is a member of a domain, and resources are centralized, you could record logon and logoff activity from the Windows Security event log only on key servers and domain controllers (DCs).

Map the high-level data requirements to the specific logs and events that provide that information. This step is perhaps the most difficult because it requires you to understand what data is available in each log.

Step 4: Identify security log and event platforms. Identify the platforms on which the security logs and sources of security event data reside. Most business environments are heterogeneous, having a mix of Windows, UNIX, Linux, Cisco Systems, and NETGEAR hardware and software, for instance. Count appliances, workstations, and servers alike. For each platform, determine whether there's a security log of interest. If the platforms don't use SNMP, syslog, or another protocol to export their logs, you need to make sure that your chosen log management solution will work on those platforms.

Step 5: Determine how you want to use the data. Do you want to analyze security data in real time—perhaps using a security event management tool—or simply collect it for analysis (to be done later or only if a particular need arises)? If you have multiple systems, each running a different OS and each with a security log, you might want a tool that lets you configure and manage the security log on each system as well as collect data.

After you've identified the security log data you're interested in, where it can be found in your environment, and what you want to do with it, compile all the information into a reference table similar to Table 1. You'll use this table when you're evaluating security log collection tools so that you'll be sure to choose a product that can collect the necessary events in the proper formats for the required realtime or batch analysis.

There are four additional questions unrelated to your security logs and the data contained within them that you must ask yourself before choosing a management tool. Security logs can contain sensitive information, so the first question is: Does the tool need to encrypt log data that it collects and moves across the network? The second question is: Does the tool need to authenticate the source of the data? If the data source isn't authenticated, an attacker could anonymously mount a Denial of Service (DoS) attack or at least spam the system with falsified data, making it difficult for you to find real data of interest admidst all the noise.

The third question is: Must the tool authenticate itself to a central collection server component, and must the server authenticate itself to the tool? Mutual authentication, combined with encryption, guarantees that the tool will supply the collected logs— which can contain sensitive information—to a legitimate server and that no one can eavesdrop on communications.

Finally, ask yourself whether maintaining the integrity of security logs during transit and in storage is a requirement. If you can't guarantee the integrity of the data, it might not satisfy your external auditors or be admissible in a court case. The answer to each of these questions is probably yes.

Security Log Management Tools
Now that you have your criteria, you can begin evaluating management tools. There are several high-quality free or inexpensive tools that you can download and use. I recommend experimenting with each in a test lab environment before choosing one and deploying it in a production environment.

No discussion of security log management tools can omit syslog. Designed as a log collection tool for UNIX systems, syslog captures security log data from UNIX and Linux hosts and appliances such as firewalls and routers but suffers from many of the security problems that other UNIX management tools do, including lack of confidentiality, integrity, and availability. Windows implementations of syslog are also available.

Request for Comments (RFC) 3195, which you can read at http://www.ietf.org/rfc/rfc3195.txt, defines a newer standard for syslog. There are other proposed extensions and standards, including San Diego Supercomputer Center (SDSC) Secure Syslog (http://security.sdsc.edu/software/sdscsyslog) and BalaBit IT Security's syslogng (http://www.balabit.com/products/syslog_ng). All proposed extensions and standards were designed to address the weaknesses of the original syslog. You need to be aware of the differences between standards when selecting products that claim to be syslog-compatible.

You already have a syslog server on your network if you're running Microsoft Operations Manager (MOM) 2005, including MOM 2005 Workgroup Edition. You can create a syslog data provider and create and configure a rule group to log syslog messages to the MOM Operator Console. Unfortunately, MOM 2005 supports only classic syslog with all its security problems. If you can wait, the upcoming version of MOM, MOM System Center 2007 (currently in Beta 2), comes with Audit Collection—formerly known as Audit Collection System— which lets you securely consolidate security event logs from Windows systems for central analysis.

If you don't have MOM, or you require a higher level of security, a great freeware syslog server is Kiwi Enterprises Syslog Daemon for Windows, which is available at http://www.kiwisyslog.com. A retail version with additional features is also available. Syslog Daemon allows incoming connections from syslog agents over TCP as well as UDP, addressing the original syslog's limitation of using only UDP. Neither the freeware nor the licensed version of Syslog Daemon addresses syslog's confidentiality problems, so syslog data will be sent across the network in clear text. You can encrypt the data sent to the syslog server using Kiwi Secure Tunnel, although the freeware version restricts you to a single secure tunnel.

Deploying a syslog server in your environment is great if you want to capture security log data from UNIX and Linux hosts and appliances such as firewalls and routers, but it doesn't help you gather data from your IIS and IAS logs, security logs generated by servers and applications, or Windows Security event logs. One option is to deploy agents that can gather data from these security log sources and convey it to a central syslog server. LogLogic's Project Lasso (http://www.loglogic.com/logforge/#) can capture the Windows Security event log and several other Windows event logs. Project Lasso sends security log data to a centralized logging server by using the syslog-ng protocol.

Syslogng doesn't offer client or server authentication or encryption, but it does use TCP instead of UDP to transport messages across the network. (You can use a third-party tool or IPsec if you want to encrypt traffic and add authentication.)

Another great tool that's based on syslog is InterSect Alliance's System iNtrusion Analysis and Reporting Environment (SNARE), available at http://www.intersectalliance.com/projects/index.html. SNARE supports multiple platforms and applications, including Windows, Microsoft ISA Server, IIS, Apache, IBM's Lotus Notes, and a variety of UNIX and Linux systems. This support makes SNARE easy to deploy, and SNARE's remote administration features make it easy to use. SNARE also comes with a server to consolidate logs from agents. However, SNARE uses classic syslog and might not be suitable for your environment due to the security problems I described earlier.

Microsoft offers a tool called Event-CombMT that can be used to collect Windows Security event logs for centralized analysis. For more information about EventCombMT, see "Take Advantage of the EventComb-MT Utility," February 2003, Instant-Doc ID 37450, and the Web-exclusive article "Collecting and Analyzing Event and System Logs," March 2006, InstantDoc ID 49492.

Another popular tool for working with the Windows event logs is Log-Parser. You can read more about this tool in "LogParser," May 2004, Instant-Doc ID 42174. And the new Windows PowerShell offers the Get-EventLog cmdlet, which you can read about in Reader to Reader: "Use Cmdlets to Monitor Your Security Event Logs," page 12.

In a future article, I'll look at products that SMBs can use if they need more features than the free tools provide.

Best Practices
Regardless of the security log management solution you choose, following some best practices can ensure that the security log data under management is as useful as possible. First, be sure you secure the repository of stored logs. I recommend that you use a dedicated server and restrict interactive and network logon access to the server.

Second, ensure that the clock on every system that has security logs from which data is collected is synchronized with a reliable time source. You have a variety of options to choose from, ranging from the very expensive (dedicated time servers on your network that are synchronized with reference clocks by radio) to the cheap (an application that checks an Internet-based Network Time Protocol— NTP—server such as time.windows .com). If your system clocks aren't synchronized, correlating events from multiple systems is extremely difficult— if not impossible.

Third, frequently revisit your decision about what security log data to manage to ensure that the data you're collecting is what you need to fulfill your compliance and security policy requirements. Also, whenever you reconfigure or add to your environment, consider the impact the change has on your security log management strategy.

Fourth, calculate how much storage you'll need for the log data you've collected. You'll need to figure out how much data is collected daily and how long you need to retain it. If you're subject to compliance requirements around retention periods, chances are you'll need to back up log data to long-term storage, such as tapes or CD-R discs, and purge old logs that are no longer necessary online. Assume worst-case scenarios to ensure that security log data isn't inadvertently lost due to storage problems.

Finally, check your log collection system frequently to ensure that logs are being collected correctly. Create a series of tests to generate representative security log data, then use that data to verify that you're collecting and managing log data properly. There's nothing worse than turning to collected logs to determine what happened after a security incident, only to find that the logs weren't collected and are now lost.

If you choose a classic syslog-based security log management tool, I recommend that you create a management network between your monitored servers and your centralized logging server. You can use a physical or virtual network. Only data related to management should flow over this network, and your security log management traffic should be restricted to this network. These practices will address potential confidentiality and availability concerns (but don't address integrity problems).

The National Institute of Standards and Technology (NIST)—part of the US Department of Commerce—recently released a draft of NIST Special Publication 800-92, "Guide to Computer Security Log Management." Some of the best practices I've described can be found in this guide. Although the guide was designed for use by federal agencies, you can use it and other special publications in the 800 series in your company.You can obtain this guide from http://csrc.nist.gov.