Taming Event Log Chaos

One difficulty of monitoring device and application events is that each device and app has its own log and often its own event format. To view all these disparate events often means collecting and sifting through each log separately. "Security Log Collection," November 2006 does a great job of stating the problem, if you're not familiar with it already.

Various entities have tried to solve this problem by proposing an event-logging standard. Most recently, eIQnetworks announced the Open Log Format (OLF), which the company described as "the industry's first open source event logging standard," saying that it "promotes interoperability that enables organizations to more easily manage and understand the log data collected from network devices, systems, and applications."

An event-logging standard is helpful only if the companies that sell the network devices, systems and applications that generate event logs support it. eIQnetworks reports that Astaro, Clavister, Cyberoam, iPolicy Networks, Secure Computing, and Top Layer Networks will use the new OLF standard. You can find out more and download the standard at http://www.openlogformat.org.

This past spring and summer, there was some noise on various blogs about Common Event Expression (CEE), a standard being developed by MITRE and the CEE Working Group. You won't easily find information about CEE on the MITRE Web site, but blogger Raffael Marty posted some information about it on his site at http://raffy.ch/blog/wp-content/uploads/2007/05/cee-r2.pdf. The PDF file says:

"The CEE Initiative recommends that the industry coordinate in four areas to facilitate log transmission and interpretation:

The Open Group has the Distributed Auditing Standard (XDAS), which it terms a "preliminary specification" and describes as follows: "The XDAS specification defines a set of generic events of relevance at a global distributed system level, and a common portable audit record format to facilitate the merging and analysis of audit information from multiple components at the distributed system level. Four groups of APIs are provided to accomplish this." Find out more about this standard at http://www.opengroup.org/pubs/catalog/p441.htm.

About a year ago, ArcSight introduced the Common Event Format (CEF), "an open log management standard that improves the interoperability of security-related information from different security and network devices and applications. ... CEF enables technology companies and customers to use a common event log format so that data can be easily collected and aggregated for analysis by an enterprise security management system." At the time, ArcSight said that AirTight Networks, CipherOptics, DeepNines, Intrusic, Reconnex, Vericept, and Vontu would support the CEF standard. Look for more information about this standard at http://www.arcsight.com/solutions_cef.htm.

For a sampling of what bloggers are saying about these standards, go to "An Auditing Standard: Has this rough beast's hour come round at last?" July 17, 2007 and "Open Log Format - What a Great Standard - Not," September 14, 2007.

The CEE and XDAS standards arguably have the best chance of eventually making security administrators lives a little easier. They're sponsored by not-for-profit organizations and are being worked on by groups of contributors. CEE in particular seems to have some energy behind it, if blog posts are any indication. Until you begin to see the effect of these standards, the following articles on our Web site can help you get your event logs and events under control.

For help collecting events in multiple formats:

"Security Log Collection," November 2006
describes practices and free tools that small and midsized businesses (SMBs) can use to gather and work effectively with a variety of event formats and event logs.

"Enterprise Event Logging for SMBs," May 3, 2007
describes six third-party log collection and management suites: GFI EventsManager 7.0, Dorian Software Creations' Total Event Log Management Suite, Engagent’s Sentry II, TNT Software’s ELM Log Manager 4.0, Prism Microsystems’ EventTracker, and RippleTech’s LogCaster for Security Auditing & Systems Management.

For help with centralized Windows event collection:

"Collecting and Analyzing Event and System Logs,"March 28, 2006
and "Take Advantage of the EventCombMT Utility," February 2003
cover Microsoft's free EventCombMT utility for gathering Windows event logs from multiple systems.

"Windows Eventing 6.0," September 6, 2007
describes Windows Server 2008's and Windows Vista's new event-forwarding capability, which lets you set up a collection server and forward events to it from other machines.