Go Spelunking for IT Data with Splunk - 24 Apr 2007

IT administrators rely heavily on systems management tools to help them identify problems in their environment. But to use the information these tools provide, you often must gather data manually, one system at a time. Splunk (http://www.splunk.com) is an IT data search engine that solves this problem by doing for IT administrators what Google did for Web users: Splunk automatically searches huge amounts of data that are generated and collected in server logs on disparate systems and platforms on your network. Two Windows IT Pro editors, Anne Grubb and Lavon Peters, spoke to Michael Baum of Splunk to discuss the product.

Splunk integrates with several systems management products such as those from CA, HP, Nagios, and IBM Tivoli. Users can launch an in-context search on an alert that’s generated by one of these products, to use Splunk to find all the machine-generated data the other products aren’t collecting. Splunk saves systems administrators countless hours from collecting such data by hand and reduces the time necessary to diagnose problems.

The company is also licensing the software to OEM vendors for use in their products. On March 12, Splunk and Proofpoint (http://www.proofpoint.com) announced that Splunk’s search technology would power the new Proofpoint Smart Search product. This messaging-security product will be able to trace inbound and outbound messages anywhere in an organization’s messaging infrastructure, to guard against any type of messaging threat (e.g., spam, viruses).

Splunk runs on Windows, every flavor of Linux, most flavors of UNIX, Sun Solaris, FreeBSD, and Mac OS X. A free version of the product is available, which can index as much as 500MB of real-time data per day—many small businesses use this version. An enterprise version is also available, with scalability from 1GB to 10TB of data per day. The price depends on the amount of data you want to index; for example, $2,500 buys a one-year enterprise license that indexes 500MB of data, the same amount as the free version, but unlocks additional features such as LDAP integration.

Large-scale applications with lots of components are Splunk’s specialty—for example, Microsoft .NET and J2EE applications running on a combination of Windows, Linux, UNIX, and mainframe. The US Postal Service Web site, for instance, runs Windows, Linux, and UNIX and uses Splunk to log about 5TB of data per day.

Splunk is currently on version 2, and that the company has a ten-year roadmap for the product. The company invites comment from its customers (and potential customers) by publishing its roadmap for the next 18 months so that community members can comment on specific features.