Lessons from the Cyber Trenches

It's 10:00 A.M., and I'm perusing event and firewall logs, looking for unexpected messages. I've done this job weekly for a client for the past two years and in that time haven't found any indications of unanticipated activity. Today is different. The security event log on the Web server has a long list of logon failures for the standard usernames Administrator and Guest and an equally long list of logon failures for usernames that don't belong to our employees. The username Snoop, in particular, grabs my attention. The logon failures have occurred for several days; they start around 11:00 P.M. and stop between 4:30 A.M. and 5:30 A.M. I view the Microsoft Internet Information Services (IIS) 5.5 server log for the previous day and discover multiple attempts to break out of IIS by running the command-prompt executable cmd.exe. Uh oh-for the very first time, something is definitely wrong.

No Doubt-We've Been Hacked
On the Web server, I notice that the network icon lights are constantly lit, which tells me that the machine is much busier than usual. This, too, gets my attention, so I respond by turning on Network Monitor. After a few minutes of logging, I examine the log and discover FTP packets in the trace. This is strange because, for security reasons, I disabled FTP on the Web server nearly a year ago. At this point, I suspect that the IIS server has been compromised.

My history of the Web server is brief. A couple of years ago, my client hired a third-party vendor to install and configure the Web server and its custom Macromedia ColdFusion application. Because I wasn't familiar with the ColdFusion software or the client's application, I incorrectly assumed that the vendor had secured the machine. I know what you're thinking right now, and you're right-this was a bad assumption. The vendor hadn't installed security hotfixes and hadn't secured IIS 5.5 with the IIS Lockdown utility. An IIS 5.5 Web server in its default state is vulnerable to innumerable known exploits, many of which can destroy the server.

After discovering how vulnerable the client's Web server is, I'm grateful that it's still running and serving up the ColdFusion application that's a key component of the company's daily business operations. As a safety precaution, I disable access to the Web server from the Internet. After I've successfully disinfected and secured the machine, I'll put it back online again.

Next, I start digging around in the Web server's Inetpub directory. It doesn't take me long to locate the rogue FTP server and the associated log files. Luckily, I find the installation log that was created when the intruder installed the FTP server-the installation log identifies the files and directories that the rogue FTP server requires and thus is a straightforward guide to disinfecting the system. As exploits go, this one isn't very sophisticated because it leaves behind a record of all its activities. The FTP log indicates that the machine has been downloading and uploading massive quantities of files for days. Some of the filenames are reminiscent of UNIX software, possibly stolen, some might be music files, and some are difficult to categorize, possibly spam.

As I explore the footprint and evaluate the impact of this intrusion, I surmise that the exploit's purpose has been to hijack the machine as an intermediate way station, a place to temporarily store files in transit from an unknown origin to an unknown destination. It's also possible that the attacker wanted to effect a Denial of Service (DoS) attack-with enough files in transit, the FTP server could potentially consume all the Web server's available bandwidth, thereby making the ColdFusion application unavailable to internal and external users. Fortunately, users can still access their critical business application, although its performance is slowed significantly by the background file transfer activity.

Using the installation and FTP transit log files as a guide, I move critical files to a safe location (for audit and future troubleshooting and disinfection purposes), delete the files and directories noted in the installation log, remove files that aren't part of the Inetpub root or the ColdFusion application, search and clean the registry of all references to the rogue FTP server, and reboot. The machine starts cleanly, and I see no obvious remnants of the FTP exploit.

Only the Beginning
Nevertheless, I'm not confident that the machine is ready for prime time; the intruder who compromised the machine might have stashed even nastier code elsewhere on the system. I search the Internet for currently active FTP exploits and locate one that exactly matches this hack, along with a description of the hack's footprint. Next, I download and run two utilities that scan for, report on, and remove known hacks and Trojan horse programs. The utility that purports to clean up this particular IIS exploit reports that the machine is clean. I give myself a pat on the back for manually expunging the rogue software without instructions. Handling this challenge was easy.

Next, it's time to secure the Web server. I run Microsoft Baseline Security Analyzer (MBSA), identify and install a myriad of necessary OS, IIS, Microsoft Internet Explorer (IE), and Microsoft SQL Server service packs and hotfixes, run the IIS Lockdown utility, and change the default SQL administrative account from sa to something more secure. I also stop and disable many unnecessary system services and enable account lockout after two logon failures to prevent malicious users from having endless opportunities to discover valid passwords for existing accounts. A second MBSA run reports the system as current on all fronts. I've spent a difficult few days but am confident now that the system is less vulnerable. Next, I change the Web server's static address, change the default HTTP port from 80 to 83, and configure RRAS to map incoming requests to the new address on port 83 to the Web server. I pass the word that the ColdFusion application is back online, and the users who usually work from home celebrate. I'm ready to sit back, put up my feet, and browse event logs that indicate all is well.

Still, I'm uneasy about the IIS hack and my preventative security measures. When I check the event and firewall logs the next day, I'm startled to discover that the Web server's event log has recorded more logon failures the previous evening, again between 11:00 P.M. and 5:00 A.M. This time, there are fewer logon-failure records because the account-lockout policy now disables malicious hackers after two failed attempts. The IIS log shows that the IIS URL Scan feature denied multiple access attempts to the ColdFusion application, some from users with IP addresses on the internal network who attempted to access the ColdFusion application during this time period.

Although I feel stupid because my earlier efforts haven't stopped the break-in attempts, I'm not slow enough to think that the employees are accessing the Web site during the wee hours of the morning. Internet-based intruders must have done this. What's frustrating is that I can't figure out how they're getting in-the firewall blocks access on most ports, the Web server has a different address, HTTP requests are mapped to a nonstandard port, and we haven't distributed the new static address outside the company.

Calling in the Reinforcements
It's time to ramp up the intensity of my defense. I return to the office around 10:00 P.M., hoping to catch the intruders in action. I'm in luck: They're trying to get in again.

My client's firewall runs on a Windows 2000 Server system with two NICs and third-party firewall software. By default, the firewall denies access to all ports unless I add a rule permitting access. I configured the firewall to permit Internet-based access on only a few TCP and UDP ports, including Network Time Protocol (NTP), HTTP requests to TCP port 83, and DNS requests to UDP port 53. The firewall also runs RRAS in router-only mode with Network Address Translation (NAT) enabled so that internal user requests to the Internet appear to come directly from the firewall. Figures 1 through 4 document my client's RRAS configuration.

Figure 1 shows the expanded NAT key in RRAS. To assign registered addresses to the Internet interface, you right-click the Internet-connected adapter (WAN in this example), select Properties, and click the Address Pool tab, which Figure 2 shows. On the WAN Properties Address Pool tab, I configure NAT to assign up to four addresses-209.38.200.105 through 209.38.200.109-to outgoing packets (I've changed the addresses to protect my client). I click the Reservations button, which displays the Reserve Addresses dialog box in Figure 3. Here, I reserve the static address 209.38.200.109 for the internal Web server. (Be aware that NAT doesn't assign any reserved address to outgoing connections.) When you reserve an address, it's natural to assume that you must select the check box that permits incoming sessions to that address. Wrong-if you check this box, RRAS will let any system connect to this address on any incoming port. After I reserve the address, I click the WAN Properties Special Ports tab, which displays the Add Special Port dialog box in Figure 4. Here, I instruct NAT to redirect external requests to the reserved static address on port 83 to the IP address 10.1.1.19, which is the Web server's internal address.

So, how do you identify intruders in real time if you don't have sophisticated network-monitoring software? When you use Win2K Server to manage remote connections, the easiest method is to right-click the RRAS NAT Internet connection-the WAN key in Figure 1-and select Show Mappings from the drop-down menu. Doing so produces a display similar to that in Figure 5, which lists all active inbound and outbound NAT connections by source and destination address, plus source and destination port type and number. Legitimate users and intruders appear as inbound connections; internal users show up as outbound connections.

It's now about 1:00 A.M. When I view the list of active NAT connections, I see two inbound sessions that have the same source IP address. I immediately restart the RRAS service, which disconnects all connections-a safe practice in the middle of the night. Just a minute later, I see a new inbound connection to a NetBIOS port, followed immediately by an inbound connection to the Web server. I have to rely on my memory now, because the screen shots I took during multiple monitoring sessions were subsequently lost when the firewall died. I carefully record the addresses of the inbound connections. After more than an hour of watching, I determine that the source IP addresses of three individual intruders were registered in France, New York, and San Francisco area. (There are many Web sites that tell you where a specific IP address is registered. My favorite one is http://www.domainwhitepages.com.) I restart RRAS multiple times to kick out the intruders, but they return as soon as the service restarts. Frustrated and tired, I disable Internet access by removing the port mapping on the NAT Special Ports tab and call it a day.

I Win the Battle...
This sordid saga of sleuthing continued for a week, during which time I discovered two very important facts. (For a summary of what I learned from the intrusion, see the sidebar, "Lessons Learned.") First, I learned that there's a right way and a wrong way to configure NAT address mapping. When you enable incoming sessions on an address reservation (as Figure 3 shows), NAT allows incoming requests on all TCP and UDP ports to that address. To restrict incoming requests to a specific static address and a specific port-a much more secure implementation-you shouldn't enable "Allow incoming sessions," and you should tweak the port numbers on the Special Ports tab. The Special Ports tab redirects a request to a specific known address on a specific port to an internal, unknown address and possibly another port. I reconfigured NAT this way, which restricted the intruders to only port 83 on the static address that I mapped to the Web server. This change cut down-but didn't eliminate-access attempts.

The second, equally important, fact was that whenever I enabled Internet access in RRAS on the Special Ports tab, the intruders were back in a minute or less. There are only two explanations for this scenario: Either someone is constantly probing the firewall to determine when the static address answers, or someone has parked a Trojan horse program on the firewall that sends a packet to the intruder when Internet access is enabled. Although the first explanation is possible, the second is much more likely. During subsequent late-night cyber wars, I discovered that an intruder had parked a Trojan horse program on the firewall (or possibly on another internal system). The Trojan horse program always sent packets to the same address in upstate New York immediately after I enabled Internet access in NAT. Unlike the FTP server exploit, this intrusion was pretty sophisticated and much more difficult to uncover.

So how did I find the Trojan horse program? After hours of experimenting, I figured out three ways to uncover the culprit: watching active NAT connections, reading Network Monitor traces, and using the Active Ports utility. Active Ports is a real-time monitor that displays all open TCP and UDP ports, the source and destination IP addresses for the connection, and the local process name (if available). You can download this utility from several locations, including CNET's download site at http://www.download.com/3000-2085-10062969.html?part=65960&subj=dlpage&tag=button.

In NAT, when you display the active connections (right-click the NAT Internet connection and select Show Mappings), you can sort the display by Inbound or Outbound connections, source or destination address, or by protocol and port. As soon as I enabled Internet access, I examined the RRAS list of active connections that

Figure 5 shows. I saw an outbound connection to the same address every time, in less than one minute. When I started Network Monitor before I enabled Internet access, I saw the same IP address in the trace every time. After I identified the IP address that was being notified each time I enabled Internet access, I added a rule on the firewall to prohibit outbound packets to that address. Doing so effectively shut down the notification, and the intruders didn't return. This rule is still on the firewall today.

... But Lose the War
I finish this discovery and repair process on a Friday afternoon, visit friends at happy hour, and celebrate that I've finally put enough barriers in place to keep the intruders out. Once again, I'm wrong. Although I'm pretty confident, I decide to check the firewall and the logs one last time on my way home. When I enter the computer room, I see that the firewall is frozen solid. I reboot the firewall. The logon screen is displayed but doesn't respond to the mouse or keyboard, so I reboot it a second time. This time, the power lights go to amber, but never green. The machine is gone for good. Subsequent attempts to reboot the firewall also failed, permanently stuck in amber like a fossil from another time.

When the firewall died, I lost many of the real-time records and screen shots I took during this cyber war. We moved the hard drive to another throwaway machine and tried to boot it up. Want to guess at the outcome? You're right, more amber lights and another dead system. Dead and gone forever.