Anatomy of a Botnet

When IT consultant David Soussan investigated a client’s network-connectivity problem, he used an Ethereal network sniffer to view network traffic entering the firewall and identified suspicious patterns. He then found the Microsoft Windows 2000 Server computer that was the source of the bad traffic, isolated a suspicious .exe file on that system, and determined that the program was a botnet connecting to a botmaster at various DNS addresses. David successfully disabled the botnet by putting it in an “isolation chamber” and blocking access to the TCP port that the botnet was using.

David Soussan got hooked on computers when he took a FORTRAN programming class in junior high. Now the owner of DAS Computer Consultants, not long ago David encountered a perplexing network troubleshooting challenge at a client site. The company's Internet access was down, and David’s initial investigation pointed to a bad firewall. However, the problem was actually something much worse: a botnet that was commandeering the company’s systems. In a recent conversation, David told me how he uncovered the worm’s trail and what IT pros can learn from his experience.

Explain the client's network problem and how you investigated it.
The client’s ISP said, "It looks like [the network is] up, but we're seeing some funny traffic." From inside the LAN, I got echoes pinging the firewall but not the WAN router. Then I tested the Internet connection by disconnecting the WAN router and firewall and attaching a single client to the WAN. The ISP was right: The Internet connection worked, so I knew the problem was related to the LAN. I reconnected the firewall to the WAN router, connected the PC to the closest switch, and still couldn’t ping the WAN. It appeared to be a firewall problem. I replaced the firewall with another product, and the LAN worked OK. But when I tested the firewall offline, it was good.

One of my favorite phrases is, "If it walks like a duck, and it quacks like a duck, chances are pretty good it's a duck." This problem looked like a duck and quacked like a duck, but it turned out to be a decoy. Even replacing the firewall didn’t solve the problem. That's when I started wondering, "Wow, what's going on here?"

I suspected that traffic on the LAN was killing the firewall, so my next step was to use a network sniffer to view the traffic going into the firewall. The initial sniff looked like a lot of junk flying around the LAN. By applying a succession of smaller and narrower filters, I found that one particular system’s traffic was suspicious because the system was attempting connections to many machines.

I disconnected that machine—a Windows 2000 server—from the LAN. The next step was to determine what on that computer was sending out the bad traffic. I rebooted the suspect system, then ran Mark Russinovich's TCPView and Process Explorer tools on it to see what programs might be opening network connections and what their traffic was doing. I started the sniffer, then very briefly connected it to the LAN long enough for it to get an IP address. TCPView lit up like a Christmas tree, showing a couple of hundred connections that instantly formed to batches of addresses all over the Internet. The program making all these connections was called ifconfig.exe in c:windowssystem32.

How did you determine that a botnet was running on the server?
When I viewed the ifconfig.exe process with Process Explorer and suspended it, ifconfig.exe claimed it was from Microsoft, but Process Explorer wouldn't verify ifconfig's digital signature. When I looked up the file in the c:windowssystem32 directory, it was marked read-only, system, and hidden. I opened ifconfig.exe by using the Attrib command, copied the file to a USB drive, and sneaker-netted the file to my notebook. When I copied the file from the USB key into my notebook, Trend Micro's antivirus software instantly flagged it as WORM_RBOT.CWU and didn’t let me touch the file.

I wanted to know what else the worm did. I put it into an “isolation chamber” built with a Windows 2000 Server virtual machine (VM) running on my notebook. With Ethereal sniffing the network and the worm copied into the VM, I watched as the worm was started. It immediately disappeared from the desktop copied into the c:windowssystem32 directory and started phoning home via DNS queries to a dynamic DNS service in Hungary. Later I ran a Traceroute, which localized the target IP address to a dialup account in Russia. The next morning, the bot client in the isolation chamber connected to its master, received orders, and started trying to exploit vulnerabilities in Symantec's antivirus product on port 2967 at various Internet locations. Three weeks later, the botnet's DNS name resolved to an address in Brazil. This told me that the botmaster was using other hacked machines to control the bots.

Had you ever encountered a botnet before?
I hadn't touched one up until that point. I knew the concept of a botnet. It's basically a remote controlled system, and you have bunches of remote controlled systems that communicate over Internet Relay Chat (IRC). That was the extent of my knowledge of botnets and what they did. But by watching the data flowing back and forth when I put the bot in the isolation chamber, I could see what it was sending out and receiving back from its controller. Once I could see what it was doing and how it was getting its command and control, neutering it was a simple next step.

How did you get rid of the bot?
Network traffic from the sniff indicated that the bot was “phoning home” on TCP port 4212. From another sniff, I found that many client machines were also compromised and running the same worm. Meanwhile, the isolation-chambered worm was still checking every few seconds for its master, with no answer. My client runs its own DNS servers, which forward requests for out-of-their-domain names to the ISP's DNS servers. To prevent the worm from phoning home, I added a DNS record in their DNS servers telling the servers they were the authoritative DNS for the botnet's domain name.

I stopped the worm on one server and watched. It didn’t respawn—evidence that it didn’t have another program watching its back. I also carefully watched the network traffic to detect whether any other attempts were made to connect to anywhere else—and saw none. Deleting the ifconfig.exe file and removing run keys appear to be a valid fix, though I was a bit cautious. We were lucky; this wasn’t a very smart worm. We opted to leave the less-critical server's worms suspended and manually cleaned the critical servers.

I also blocked all access to port 4212 on the firewall. I knew that copies of the worm were still running on client machines. When I viewed the traffic on port 4212, I compiled a list of infected machines by IP address; the client’s IT staff could deal with those. At this point, all the servers were either cleaned or the worms on them suspended.

What can IT people learn from your experience?
The network was infected because the company was running an unpatched version of Symantec’s antivirus software. There was a patch for the vulnerability that infected this computer, but this company’s IT staff didn't apply that patch and roll it out. The bot came in via a notebook that was used offsite, probably connected to someone’s home DSL and unprotected by a firewall, and brought back into the corporation. So the lessons are, convince users who take data offsite to be more careful and vigilant, and make patch management a priority. An unpatched system is like a wide-open door. You’d fix a broken outside door in your home, and you need to treat your corporate assets—and whatever’s protecting you from malware—the same.

An IT person needs to know what's happening on his network. You have to get a protocol analyzer, and you've got to learn what the protocols are and how they work. And you've got to learn enough so that you can recognize the kinds of things that aren't good. That was one of the reasons for the Webcast that I did for Microsoft on the art of debugging, using a network sniffer to isolate problems. (For more information, see "TechNet Webcast: The Art of Network Debugging (Level 200)," May 15, 2007, at http://www.microsoft.com/events/webcasts/library/200705.mspx.) Once you see these kinds of things and see what's out there, the world kind of opens up, and you’re thinking, "Wow, I had no idea that's how that happened."

You can diagnose a lot of problems by watching that network traffic. You can tell whether the problem is something local to a system or something happening over the network. If you're going to debug or figure out what's going on or design a system that relies on more than one of these components, you've got to have knowledge that spans those components. The network is usually what spans them. In this industry, if you're not learning, you are falling behind very quickly. I don't want to fall behind. I want to keep growing, and keep learning, and keep doing new things.

Have you found that companies are doing a good job with their network security, or is this a deficiency?
We do what we know we need to do most of the time, but we're not diligent or vigilant about it, because there are always other fires to put out. Security is about spending money and effort and time to make sure nothing happens. How much should a company spend on its internal security? Where's the magic line where you've spent enough to make sure nothing happens? How do you convince the CFOs who don't understand the level of vulnerability that's there? How do you get them to say, "You know what, this is a good investment. We should spend it"? How do you get them to spend money to make sure that nothing happens?