Large-Scale, Unstructured Data Retrieval and Analysis Using Splunk

When it comes to processing massive datasets, the implementation of MapReduce is the performance and scalability model for parallel processing executed over large clusters of commodity hardware. Out of MapReduce, several languages and frameworks have evolved: Google Sawzall, Yahoo! Pig, the open source Hadoop framework and Splunk.

While MapReduce is an essential element to scaling the capabilities of search and reporting in Splunk, the out-of-the-box benefits of using Splunk for large-scale data retrieval extend beyond MapReduce processing.  

Unlike other MapReduce languages and frameworks that require custom scripts or code for every new task, Splunk utilizes its search language to automate complex processing. The Splunk Search Language makes challenging data analysis tasks easier without requiring the user to control how it scales.  Beyond the simplicity of the search language, Splunk provides a universal indexing capability to automate data access and loading. With numerous mechanisms for loading data, none of which require developing or maintaining code, Splunk users are productive quickly.