You've Been Hacked—Now What?

Recovering and learning from attacks

Whether it's the phone ringing at 4:30 a.m. or your pager going off while you're in a restaurant, it's the news no one wants to hear—your company has been attacked, and everyone is turning to you for guidance and instructions. What do you do now? If you have a disaster-recovery plan in place with a contingency for this situation, you can turn to it for direction. If you don't, read on. Even if you do, read on: You can never be too prepared.

First, don't panic. This advice is much easier to give than to follow, but keeping a level head helps not only you but also your organization recover from what can be one of the most traumatic and embarrassing experiences it can suffer. What you do in the minutes, hours, and even days following an attack can have a lasting effect on not only your organization but your career. In short, making the right decisions is crucial.

Assessment and Immediate Response
Before you make any decisions, fully assess the situation and the damage inflicted. Start by auditing systems to see which were affected, then take steps to isolate those systems from the rest of your network. Pulling out the network connections is the fastest and surest method, but don't power the systems down: You'll need access to them to determine how they were attacked, and you might lose crucial evidence if you reboot them. By disconnecting your servers from the network, you prevent the intruder from returning. Disconnection can also contain the problem by reducing the likelihood that a virus, worm, or other malicious code will spread from the affected systems to other systems in your organization.

For each affected system, document the installed OS versions, applications, service packs, and hotfixes, then make image backups. An image backup is a backup that includes unused disk sectors and deleted files. Later, with the right tools, you can examine these backups for hidden files and evidence the intruder left. Image backups can also be invaluable to law-enforcement personnel if the incident results in criminal prosecution.

Depending on the nature of the attack, you might need to take immediate action. For example, if your Web site was defaced, replace it with a simple page that informs visitors that the site is down for maintenance. If your Web servers sit in a load-balanced or a DNS round-robin array, each with a local copy of the Web site, you can reconfigure the array to take the affected machine out of service and let the remaining machines handle the increased load. Taking such an action can be risky because the vulnerability that let the intruder deface your site on one server probably exists on all the other servers. If the attacked system is a central server, such as a file and print server or a database server running Microsoft SQL Server, your safest course of action is to disconnect that machine and the associated infrastructure (e.g., Web and application servers). If database or file replication takes place between the affected server and other servers, pause the replication until you know what data, if any, the intruder modified. Also, stop any batch and scheduled jobs or system services that manipulate data, and make sure they don't start if you reboot the server. Keep a list of such processes for each server in your data center, along with details about how to shut down the processes in an emergency.

At this point, you have two conflicting goals: restoring service to users as quickly as possible and determining the damage and how the intruder broke into your systems. By taking steps to restore service, you might erase the audit trail and evidence that can help you identify what was damaged, what exploit the attacker used, and perhaps even who the intruder is. Without knowing what was damaged, you can't repair it. Without knowing how the intruder gained access, you can't prevent another attack. The safest way to proceed is first to determine how the attacker penetrated your defenses and what damage was done. Only then should you restore the attacked systems and place them back in service.

Analyzing the Attack
In determining how the intruder accessed your systems, check for the usual flaws and weaknesses. Here are steps to take to check for weaknesses and to minimize your risk of attack:

Although this list isn't definitive, it represents the majority of vulnerable areas that intruders exploit.

If your systems are accessible from the Internet, audit your firewall configuration to verify that you've granted access to only those servers in the demilitarized zone (DMZ) intended for public access and to those specific ports required to provide service (e.g., FTP, SMTP, HTTP). Make sure servers in the DMZ can't access systems internal to your organization. If the attacked system wasn't accessible from the Internet, you need to conduct this audit to rule out the possibility of an external attack.

Finding vulnerabilities can be like trying to find needles in a haystack, so consider investing in a vulnerability scanner. Many commercial and public domain scanners exist. Internet Security Scanner (http://www.iss.net) and Nmap (http://www.insecure.org) are a couple of examples. Use a virus scanner with an up-to-date signature file to detect viruses and Trojan horses that intruders use to access your systems. Currently, no one product can find and identify all potential vulnerabilities or weaknesses.

Unfortunately, following the checklist I've given you and running a scanner aren't always enough to determine how an intruder accessed your system. If this is the case, you can examine the audit trail. Check for suspicious user activity, running processes, and executables that the attacker might have used or left behind. A common intruder goal is to secure a means of coming back into an attacked system with Administrator-level access. Intruders often secure this access by creating accounts that are members of the Domain Administrators group. Intruders are increasingly using Trojan horses and back doors such as NetBus and Cult of the Dead Cow's Back Orifice. Rootkits, the bane of UNIX systems administrators, are beginning to make an appearance, too.

If you've properly configured event logging, the Security log has details about who logged on to the system, when, and how (i.e., interactively, over the network, or as a service). Figure 1 is an example of Event Viewer showing a user logon event. The Security log can also track process creation and attempts to access files and registry keys. Check for suspicious activity from user accounts that don't ordinarily access the affected systems, activity from accounts that belong to employees on leave, and activity outside typical business hours. These symptoms often occur with a hijacked account. Also, check for new user accounts that don't appear to belong to valid users.

Check your Microsoft IIS logs for suspicious activity, especially if you use Perl or other interpreted languages for Common Gateway Interface (CGI) scripts. Often, a poorly written script can let an attacker into your otherwise locked-down server. Also check other logs, (e.g., the DHCP logs) for suspicious activity.

You can use the ps.vbs utility (available in the Windows NT Server 4.0 Resource Kit) to enumerate running processes, as Figure 2 shows. The ps.vbs utility is superior to Microsoft Task Manager because it shows the full path to the executable file and can show processes on a remote machine. Running the Netstat command from a command line can reveal suspicious activity by showing network connections to your system. If you see any suspicious processes or activity that you don't recognize, investigate further because an intruder might have installed a back door on your system. You can find a list of TCP/IP ports that Trojan horses and backdoor processes commonly use at http://www.sans.org.

You can also run the System File Checker (SFC—sfc.exe) utility, which is part of Windows File Protection (WFP), to check system executables and DLLs for corruption or tampering. A common intruder technique is to hide a system DLL, then copy in a Trojan horse that has the name of the original DLL but uses the hidden DLL to fool the user into believing nothing has changed. You can use commercially available and public domain Intrusion Detection Systems (IDSs) such as Tripwire to detect changes to files and registry settings and to define what to profile and scan for changes. If all else fails, use the Search option from the Start menu to find files created or modified in the last n days. Check the registry for changes to entries that control process creation at startup or logon and entries that load and run code in DLLs in reaction to events such as user logon, password changes, and network activity. Remember that intruders often use legitimate-sounding filenames to hide their activities, so examine each file and registry entry carefully.

Restoring Service
Because backups might hold previously compromised executables and data, best practices dictate that you restore the OS and applications not from a recent backup but from the original installation medium, followed by the appropriate service packs and hotfixes. Note that it's insufficient to use tools such as SFC or the installation repair tool that's part of the installation process for Windows 2000 or NT to detect and replace compromised system files. These tools aren't thorough and check only system executables and DLLs, and in some cases, intruders can circumvent them. The intruder might have modified .ini files, the registry, the SAM database, Active Directory (AD), the IIS metabase, or one of many other components that make up your installation.

After you rebuild your system, restore the data. If the data is static, such as Web pages and CGI scripts, restore it from a safe source. If the data is dynamic, such as data files or a database, validate the data before you restore it. To check for tampering, use Event Viewer to examine the security log to see who modified what and when and to search for files created or modified in the last n days.

Before you reconnect your systems to the network, run a virus scanner and change all administrator and service account passwords. Then, require all users to change their passwords at their next logon.

Disclosing Details of the Incident
The technical aspects of dealing with an attack often pale in comparison with the problems of disclosing what happened. Many organizations choose to keep the incident quiet, preferring not to alarm customers, shareholders, and partners. These days, however, intruders delight in sharing their exploits with their peers in chat rooms and on mailing lists. From the time the attack occurred to the time you discover and handle it, many people could have visited your system to confirm the attacker's claims. Some Web sites, such as http://www.attrition.org, take screen shots of attacked Web sites and publish them. A more problematic scenario is when the attacker publishes data from your systems, such as customer credit card information, on the Internet.

The safest course of action, especially when you can't assess the full extent of the damage and you suspect or know that the intruder has compromised confidential or sensitive data, is to inform all parties (e.g., customers, shareholders, partners). Tailor the information you provide each group. For example, customers and credit card companies need to know that card details have been compromised, partners need to know how the attack was carried out so that they can secure their systems from a similar breach, and shareholders need to know the financial impact of the incident and the steps you're taking to prevent another attack.

When dealing with the press, appoint one person to act as spokesperson for your organization. Make sure no one in your organization leaks details about the attack method, especially if you've yet to secure other potentially vulnerable systems or inform partners.

Finally, consider documenting the incident and using it as a training tool. At the very least, use it to verify that your disaster-recovery plan actually worked. If you don't have a plan, use your response to the incident as the starting point to draft one.