Toolbox: Grep

Keeping your network secure involves a number of day-to-day tasks. One of the most important is keeping tabs on your systems' logs. Quick retrieval and interpretation of log data during a security event or other alarm is crucial. And quality security software and hardware almost always generate events or logs that document use (or attempted misuse) of various programs, services, and systems, making log analysis a vital skill.

Windows system logging uses the Event Viewer, but many other built-in Windows programs, such as Microsoft IIS and Internet Authentication Service (IAS), log to text files. A number of programs let you stream Windows logs into a text file (in case you've fallen in love with text-based log management), and let's not forget syslog, which lets you aggregate remote systems' logs to a common syslog server that stores those logs as text files. Two venerable log-analysis tools--grep and tail--can be key additions to any security toolkit. The simplicity of these command-line utilities belies their extreme usefulness, flexibility, and power. Both tools are widely used by UNIX administrators, but you don't need to be a UNIX guru to immediately take advantage of either tool in a Windows environment. Grep and tail are available on most UNIX systems, and you can download GNU-licensed Win32 versions from the Sourceforge Web site (http://unxutils.sourceforge.net). Download the UnxUpdates.zip file, which contains several useful UNIX tools for Win32, then extract grep.exe and tail.exe to your computer.

Next month, I'll discuss the tail utility, but let's begin with grep. You can use grep to compare each line of a text file against a pattern-matching expression and to take action, such as displaying a line, based on a successful match. By using grep with discrete pattern matching, you'll be able to target in on the data that interests you, regardless of the size of the source text file.

Using Grep with Regular Expressions
As an example of how to use grep, let me show you how you can use the tool to find out whether a specific IP address has accessed an IIS server. IIS writes log data to multiple text files (the default is one file per day) in the %windir%system32logfilesW3SVC1 directory. Suppose you want to find all the connections to the IIS server from a computer that has the IP address 192.168.0.100. Copy grep to the IIS server, open a command line, and type

grep "192.168.0.100" *.log

This command searches through all text files that have the .log extension. If grep finds text matching the specified pattern (i.e., 192.168.0.100) in any of those files, it displays that line of text.

One of grep's most powerful features is its support of regular expressions, an extremely flexible syntax based on metadata and constructed to match specific patterns of text. For example, if you want to parse your IIS log files to look for directory-traversal attempts, which IIS denotes in its logs as ../ or .., run grep with the following syntax:

grep ..[/\] *.log

This example shows how to construct a pattern that uses metacharacters as forced literals. Typically, a regular expression pattern uses the period (.) to represent any character; for example, the regular expression

h.t

would match hot, hat, or hit. To find a literal period, as in our example, you need to escape the character by using the backslash metacharacter ():

.

To find a double period (..), you need to use the pattern

..

To find a forward slash (/) or backslash, you need to use square brackets ([ ]), which tell grep to look for any of the characters within the brackets, similar to using an or clause. For example, the pattern

h[oa]t

would match hot or hat but not hit. Furthermore, because the backslash is a metacharacter, you need to escape it as you did the periods:

\

Therefore, the regular expression needed to find ../ or .. would be

..[/\]

Regular expressions make use of several metacharacters, which you can arrange in a dizzying number of configurations to create nearly any pattern you need. (An Internet search for regular expressions will turn up hundreds of documents and manuals about how to create regular expression patterns.) JavaScript, Perl, VBScript, the Virtual Interface (VI) text editor, and many UNIX programs support regular expressions, and you can't go wrong in learning some basics of this powerful pattern-definition nomenclature.

Piping Grep
Beyond using grep to search text files, you can use it to redirect the output of another command-line program into grep. For example, the command

dir | grep DLL

displays a list of all the DLL files in a directory, as Figure 1 shows. Note, however, that Grep is quite literal in its patterns, and the previous command tells the tool to look only for the uppercase pattern. To find all DLL files (both .DLL and .dll), you need to use grep's ignore case (-i) parameter.

Let's take a look at another example:

dir C: /s | grep DLL -i -c

This command will return the number of DLLs on the local system. First, the command runs the Dir command, starting at the root of the C drive and using the include subdirectories (/s) parameter to search the entire local system. The command then pipes the output into the grep command, using the simple pattern DLL and using the -i parameter to find DLLs regardless of whether the files' extensions are uppercase or lowercase. The command then uses grep's -c parameter to count the number of matching lines.

Learn More
Most UNIX systems (and the Win32 GNU) have their own, slightly different versions of grep, so I suggest you check out the command syntax for the version you end up using. To do so, issue one of these commands:

man grepgrep -?grep /? grep --help

Next month, I'll tell you about tail, which displays the last few lines of a text file and can keep you informed of any new entries in a log file.