Building and Using an Incident Response Toolkit, Part 1

Computer security incidents (usually security breaches) are a fact of life that many systems administrators must deal with at some point. Whether threats originate from malicious intruders, self-propagating worms, or trusted insiders, the need to quickly assess and appropriately respond is vital. In addition, performing an in-depth analysis (i.e., computer forensics) of the compromised machine later is crucial. This series, which is broken into two parts, covers both types of analyses. Part 1 and Part 2 of "Building and Using an Incident Response Toolkit" discuss how to build and use an incident response toolkit so that you can quickly collect data about that incident. Then, in upcoming issues, Part 1 and Part 2 of "Performing Forensic Analyses" will discuss how to perform a thorough analysis of a compromised machine, with an emphasis on possibly bringing evidence to court.

Whether you're using an incident response tool or performing an in-depth analysis, you need to keep in mind two basic principles. First, if the incident ultimately results in criminal proceedings, demonstrating that you haven't changed the evidence in any way is important. Changing the system as little as possible and hashing acquired data is standard practice. In addition, you should document all your steps and preferably have at least one other set of eyes to vouch for the validity of your collected data. If you don't take these precautions, proving that the intruder caused the damage to the system is difficult.

Every file opened, process run, or segment of memory accessed changes the state of the machine and can impede your analysis. How to minimize change is a topic constantly under debate, and no single right answer exists. Based on the situation, the smartest move might be to immediately shut down the machine (i.e., a clean shutdown), literally pull the plug (i.e., a dirty shutdown), disconnect the machine from the network, or do nothing at all. In all cases, you lose some crucial information.

The best action is one that preserves the most information while minimizing risk. Typically, that action is to unplug the computer from the network, image the volatile memory, then shut down the machine. Your organization might have guidelines specifying what to do. The US Department of Defense (DoD) guidelines, for example, tell you to immediately pull the plug on the machine. For the purpose of this article, I'm going to discuss how to perform analyses on a live machine.

Second, you must never trust the machine. If an intruder has broken in and achieved Administrator rights, there's no limit to what might be booby-trapped or corrupted.

Because you can't trust a compromised machine, you should prepare an incident response toolkit that contains trusted copies of necessary analysis tools. You need one set of tools, which I cover in this article, to quickly analyze the compromised machine. You need a second set of tools, which I'll cover in "Building and Using an Incident Response Toolkit, Part 2," to quickly analyze the file system behind the compromised machine. Creating this toolkit before a problem occurs will save you a lot of time and headaches later.

Assemble the Toolkit
The incident response toolkit contains numerous tools from many sources. All the tools are either free downloads or come prepackaged with the Windows OS. The entire collection of tools will easily fit on a CD-ROM (business card or regular sized) or other removable disk. (A regular 3.5" disk likely won't have enough room, but if you do burn these files to a 3.5" disk, be sure that you set the write-protect setting on your medium.) You should place all the tools in the CD-ROM's root directory. To assemble the toolkit, follow these steps:

1. Dig out a trusted version of Windows. Many of the tools you'll need come prepackaged with Windows, but having separate copies of these tools is essential to protect against corrupted binaries. Therefore, dig out your installation media (or access a well-trusted system), go to the %windowsroot%system32 directory, and copy the following binaries: arp.exe, at.exe, cmd.exe, dir.exe, doskey.exe, edit.exe, find.exe, findstr.exe, hostname.exe, ipconfig.exe, nbtstat.exe, net.exe, netstat.exe, nslookup.exe, notepad.exe, ping.exe, regedt32.exe, route.exe, share.exe, and tracert.exe.

All these tools are standard diagnostic tools on a Windows machine. I won't cover them all in this article, but because they're standard Windows tools, you're likely already familiar with them. Their presence in your toolkit ensures that you can collect accurate data, whether that involves following the procedures I outline here, looking through the registry, or investigating DNS names.

2. Add ActiveState's Md5sum program, which is available at http://downloads.activestate.com/contrib/md5sum/Windows. Md5sum gives a unique cryptographic hash for each file. For example, the cryptographic hash for the md5sum.exe file might look like eb574b236133e60c989c6f472f07827b md5sum.exe. A good practice is to create a hash for each file in your toolkit, then store those hashes in a file on a separate secure medium. Another good practice is to create two hashes of any file that contains data from the compromised computer. You should first hash the file immediately after you create it, then hash the file again after analyzing its contents. By comparing the two cryptographic hashes, you can discern whether any change has been made to the file, even if only 1 byte has changed.

3. Add Netcat (a networking utility) or Cryptcat (an enhanced version of Netcat). Netcat is often referred to as the systems administrator's Swiss army knife. For example, you can use Netcat to open a listening port or send data to a port. Another common use (which I discuss here) is to send data across the network to a secure machine, where Netcat writes the data to the disk. Cryptcat performs the same functions as Netcat, except that Cryptcat encrypts data with the Twofish algorithm to prevent prying eyes from reading the data. Considering that an intruder can easily plant a sniffer on a network, I suggest that you use Cryptcat instead of Netcat because of Cryptcat's added security.

Netcat is available at http://ftp.uni-kl.de/pub/windows/cygwin/release/netcat or http://netcat.sourceforge.net and requires Cygwin (a Linux emulation software kit) to run properly. Cryptcat is a native Windows binary, so it doesn't require any emulation. You can find Cryptcat at http://farm9.org/Cryptcat/GetCryptcat.php or http://sourceforge.net/projects/cryptcat.

4. Add several forensic tools—Fport, BinText, and the Forensic Toolkit—that Foundstone offers at http://www.foundstone.com/resources/freetools.htm. Fport reports all open and listening UDP or TCP ports and provides the process identifier (PID), name, and path of the processes that use those ports. You can use Fport to quickly identify suspicious programs on the compromised machine. BinText searches through files for ASCII and Unicode strings, which can help you discover binaries that have been compromised by a Trojan horse. The Forensic Toolkit contains a useful collection of tools for dealing with NTFS.

5. Add a series of SysInternals utilities. If you go to http://www.sysinternals.com/sitemap.shtml, you'll find links to the following utilities:

6. The ntsecurity.nu Web site offers several security tools for Windows. Go to http://www.ntsecurity.nu/toolbox and download the following tools:

7. Add LADS, which is available at http://www.heysoft.de/frames/f_sw_la_en.htm. This tool lets you find hidden data streams on NTFS partitions.

8. Download the Forensic Acquisition Utilities from http://users.erols.com/gmgarner/forensics, then add dd.exe to your toolkit. (The download also includes Md5sum and Netcat, but you don't add these tools because you've already added them.) You'll want to ignore the instructions about how to organize the Forensic Acquisition Utilities and instead place dd.exe in your toolkit's root directory. Otherwise, you'll have problems using the toolkit. Originally a UNIX tool, dd.exe makes binary copies of files. You can use this functionality to image the compromised computer's physical memory to a file for analysis.

9. Add pclip.exe, which you can find in the UnxUtils.zip download from http://unxutils.sourceforge.net. Pclip.exe, which copies clipboard data from the command line, is a Windows-ported version of a UNIX program. Such tools can be helpful because they provide features not found in Windows tools.

Decide on a Storage Medium
After you add all the tools to the security toolkit, you need to decide on a medium on which to write the data you're collecting with those tools. Possible options include using USB drives, 3.5" disks, Cryptcat, or Netcat. Each option has its advantages and disadvantages.

For example, USB drives and 3.5" disks are easy to use. However, when you insert a USB or 3.5" disk, it makes changes to the system (the USB disk more so than the 3.5" disk). In addition, USB and 3.5" disks might not have enough space to save all the information, especially when you store imaging memory. With Netcat and Cryptcat, storage space isn't a problem. However, Netcat and Cryptcat require a network connection to another computer, which might not be desirable if you want to isolate the compromised computer from the network.

The Initial Response
Now that you've assembled the toolkit and you've decided on a secure medium on which to store your data, you're ready to use the tools. When a security incident has occurred on a computer, insert the toolkit disk in that machine. Open the toolkit's copy of cmd.exe by clicking Start, Run, then entering the full path to that copy. Let's say that you have the toolkit on a CD-ROM in the D drive. Thus, you'd enter

D:cmd.exe

as the path. DOS's default behavior is to search for an executable in the current working directory before searching other paths on the computer, so entering the full path is important. If you just enter cmd.exe, the compromised computer's copy of cmd.exe will likely open.

A sound forensic practice is to document every command that you run in D:cmd.exe, then hash the commands and their results to prevent later accusations of tampering. Therefore, you need to save each command's output to one or more files on your secure medium.

If you want to store the command output in a file on a USB or 3.5" disk, simply insert the disk in the appropriate drive on the compromised machine. Then, after each command, place the >> redirection symbol followed by the name of the file in which you want to record the output. For example, to write the Date command's output to a file called command_output.txt on a USB disk in the J drive, you'd type

date /t >> J:command_output.txt

If you want to write the Date command's output to a 3.5" disk in the A drive, you'd type

date /t >> A:command_output.txt

Make sure that you always use the >> redirection symbol (which appends data) rather than the > redirection symbol (which overwrites data). Otherwise, you might end up obliterating the file's previous contents.

If you want to use Netcat to store data on a data collection, or listening, server, you need to follow a different procedure. To set up Netcat so that it writes the command output to a data file called command_output.dat, run the following command on the listening server:

nc -L -p 5000  >> command_output.dat

(Although this command appears on two lines here, you would enter it on one line in D:cmd.exe. The same holds true for the other multiline commands in this article.) In this command, -p 5000 specifies the port on which to listen and the -L option tells Netcat to keep listening until you press Ctrl+C to kill the listening command (i.e., close the socket). After you set up Netcat on the listening server, you can send data from the compromised machine to the listening server by using a command such as

date /t | nc 192.168.1.100 5000

In this command, the pipe symbol (|) tells Netcat to send the Date command's output to the listening server that the IP address identifies (in this case, 192.168.1.100 5000). To close the socket, press Ctrl+C. If you want the socket to automatically close after the output is written to the listening server, you can use the command

nc -e "date /t" 192.168.1.100 5000

The Cryptcat commands are similar to the Netcat commands, except with Cryptcat, you'll probably want to use the -k option to change the enciphering key. (The default key is metallica.) On the compromised machine, you use a command such as

date /t | cryptcat  192.168.1.100 5000  -k "our_little_secret"

Note that a bug in Windows NT prevents Cryptcat's -e option from working properly, so don't try to automatically close the socket if you're running NT. On the listening server, you use a command such as

cryptcat -L -p 5000  -k "our_little_secret"  >> command_output.dat

With the toolkit's copy of cmd.exe open and the recording mechanisms in place, you're ready to start using the tools in the toolkit to obtain information about the compromised machine. To do so, perform the following steps. Note that, for the sake of brevity, the sample commands in these steps don't include the code that writes the output to the secure medium. When you enter these commands, you must include this code.

1. Capture the current date and time for documentation purposes. Run the commands

date /ttime /t

2. Use PsInfo to capture the local system information, such as OS version and uptime, for documentation purposes. Run the command

psinfo.exe

3. Use ipconfig.exe to obtain the IP addresses and network data for the host for documentation purposes. Run the command

ipconfig /all

4. Use PromiscDetect to find out whether any network adapters on the host are running in promiscuous mode. Run the command

promiscdetect.exe

As Figure 1 shows, PromiscDetect provides a warning when it finds an adapter running in promiscuous mode. PromiscDetect also warns you when it's unable to open an adapter.

5. Use arp.exe to get a list of computers that have recently talked with the compromised machine. Without going into too much detail, here's how computers communicate with one another in an Ethernet environment: Whenever a machine uses TCP/IP to communicate with another machine, the communicating computer needs to obtain the media access control (MAC) address, or Ethernet address, of the computer with which it wants to talk (i.e., the target computer). To do so, the communicating computer uses the Address Resolution Protocol (ARP) to request the MAC address from the target computer. The target computer caches such requests for a short time to save bandwidth. Thus, the ARP cache contains the machines the target computer has communicated with in the last few minutes. The ARP cache includes the IP address and MAC address of the communicating computers but doesn't include any information about what was communicated. Although the OS flushes the ARP cache about every 2 minutes and whenever you reboot, you might be able to glean some information if the crime scene is fresh. To use arp.exe, run the command

arp.exe -a

6. Use netstat.exe to check the routing table. An intruder who has already subverted a machine might be forcing other computers to route packets through the subverted machine so that he or she can scan through the packets. To use netstat.exe, run the command

netstat.exe -r

Familiarity with your network is necessary to analyze the command results. Only by knowing what is ordinary can you spot anything unusual. In addition, you must consider that the machine itself might be forwarding packets.

7. Use nbstat.exe to check NetBIOS. You should run the nbstat.exe command three times, each time with a different option: -c (which lists the contents of the NetBIOS name cache), -s (which displays client and server sessions), and -n (which lists local NetBIOS names). So, for example, the command to obtain the cache contents would be

nbtstat.exe -c

Again, familiarity with your network is necessary so that you can spot anything out of the ordinary. If the machine has a large number of NetBIOS connections, especially to nonservers, you might have a problem. Noting the connections can also help you track down other compromised machines.

8. Use PsFile to see which files are open on the network. Run the command

psfile.exe

Print any files that are open.

9. Use Fport to look for any suspicious ports. Run the command

fport.exe

Figure 2 shows sample Fport output from my workstation. Familiarity with your network will be of the utmost importance when looking at Fport results. Fport lists each process's listening port, but unless you know which ports to expect, the information is of little use. Typically, a Windows workstation will be listening on ports 135, 137, 138, 139, and 445. I also happen to have Mozilla running on ports 1034 and 1035 and AOL Instant Messenger (AIM) on port 5180.

You can find a list of ports and their associated uses in %systemroot%system32driversetcservices. However, just because the services file lists a port doesn't mean that a legitimate application is using it. Intruders can hide Trojan horses under the guise of a legitimate application. You can find a list of Trojan horses and their ports at http://www.simovits.com/trojans/trojans.html. If you click a name of a Trojan horse, you'll get detailed information about it, including filenames, what actions it takes, and where it hides in the registry.

10. If you see an interesting port in the Fport output and you want to analyze the application that has opened it, you can use the PMDump utility. The PMDump command takes two arguments: the PID of the process you want to investigate and the name of the file to which you want to dump the process's memory contents. For example, to copy, or image, the memory of the AIM process that Figure 2 shows to a file named aim_binary.img, you'd use the command

pmdump.exe 1156 aim_binary.img

Note that no easy way exists to directly capture the data and transmit it with Netcat or Crypcat.

11. If you're seeking a more thorough examination of the memory contents, you can use dd.exe to image all physical memory on the machine. This imaging takes as much space as you have memory. The dd.exe utility in your toolkit has been modified to work with the physical memory on Windows machines. The dd.exe command takes two arguments: if=filename (where filename is the name of the input file) and of=filename (where filename is the name of the output file). To image only the memory, you use \.PhysicalMemory as the name of the input file. For example to image the compromised computer's physical memory to a file called output.img, you'd run the command

dd.exe if=\.PhysicalMemory  of=J:output.img

You should use Md5sum to create a hash of the output file before and after analyzing the contents to ensure the output file's integrity. To create the hash, run the command

md5sum.exe output.img

To analyze the memory contents in the output file, you can use the Strings utility

strings.exe output.img

or

bintext.exe

to output all ASCII strings. BinText is a GUI program. From the starting screen, choose Browse, then choose Go to see the strings. As Figure 3 shows, you can also use BinText to search for a particular string. Although combing through the memory takes time, it can help diagnose problems. Intruders often use stock tools, which do little to hide their inner workings. Thus, finding strings such as rootkit, which indicates the presence of a Trojan horse copy of system binary, isn't uncommon.

Imaging the swap file, which is typically called pagefile.sys in the root directory, can also prove valuable. You can use the dd.exe program to image the swap file, then hash the output file for authenticity reasons. However, you might not want to analyze the contents during the security incident's initial investigation because of the large size of swap files.

12. Use PsLoggedOn to obtain a list of all the users currently logged on to the compromised computer. Run the command

psloggedon.exe

Again, familiarity with your network and systems is important to spot anything out of the ordinary. In particular, look for anyone logged on through resource shares. You would expect users to be accessing resource shares on a file server but not a dedicated Web server.

13. Use PsList to obtain a list of all processes currently running. Run the command

pslist.exe

In the output, you want to look at the second column, which contains each process's PID. If you find a suspicious PID, you should use PMDump to image the process's memory, then analyze the contents. ListDLLs is useful for checking a suspicious process because you can use it to display the DLLs that the suspicious process loaded. You can use the listdlls.exe command with either the PID or the name (even a partial name) of the process as the argument. For example, to display the DLLs that the AIM process loaded, you can use the command

listdlls.exe 1156

You can even use ListDLLs to list all the applications that have loaded the DLL you specify. For example, if you want a list of all the applications that have loaded msdll.dll, you can run the command

listdlls.exe -d msdll.dll

To determine which files a suspicious process has opened, you can use the Handle utility, which is similar to UNIX's lsof command. You use the handle.exe command with the name of the process as the argument, such as

handle.exe -p AIM

Although using a partial name is acceptable, using a PID is not. If you don't specify any arguments, Handle outputs information for all the processes.

14. Use Autoruns to list all the programs and DLLs set to run at start up. This information can be useful for digging up backdoor programs. Run the command

autoruns.exe

15. Use PsService to get a list of all the services running and the state they're in. You can sometimes find evidence of a Trojan horse or a policy breach. This tool can also help identify the cause of a break-in, such as a vulnerable service that was running. Run the command

psservice.exe

16. Use pcclip.exe to retrieve the contents of the clipboard in case it contains any useful information. Run the command

pclip.exe

17. At this point, you've collected all the initial data you need from the compromised machine. Now you need to use doskey.exe to create a list of all the commands you've run. Use the command

doskey /history

18. Again, capture the current date and time by running the commands in Step 1.

19. Use Md5sum to create a hash for each of the files in which you collected the command output. (You might have already created a hash for some of the files.) Store these hashes in a separate file, and print a copy of the file so that you can later verify that the evidence you collected wasn't tampered with.

The Collection Process Continues
At this point, you've gathered as much information as possible from the compromised machine. However, you're not finished yet. You still need to gather information from the file system. I'll show you how to use your toolkit to do just that next month in "Building and Using an Incident Response Toolkit, Part 2."