Protect Your Network from Intrusion

Strengthen your network's defenses with a network IDS

When I think about intrusion detection, a modern paraphrase of an old question comes to mind: "If an attack occurs on your network and no one knows about it, did the attack really occur?" Detecting attacks on your network is crucial, but doing so is also difficult. That's where intrusion detection comes in. Intrusion detection is important, especially in a multilayered defense-in-depth strategy. (For information about the defense-in-depth strategy, see the National Security Agency—NSA—paper "Defense in Depth" at http://nsa1.www.conxion.com/support/guides/sd-1.pdf.) I introduce you to network Intrusion Detection Systems (IDSs) and briefly discuss Snort, an excellent open-source network IDS that you can deploy on machines running Windows 2000 Server. I also review some network basics, including the challenge that Virtual LANs (VLANs) present, as a preface to discussing the placement of your network IDS, which is essential to the IDS's success. Finally, I review how to counter some techniques aimed at evading your network's intrusion-detection mechanisms.

Security and Depth: Layers of Protection
In a security context, depth in network protection usually implies a multilayered approach to network defense similar to the many lines of defense with which castles were protected. On the negative side, both castles and networks have some basic vulnerabilities: Both are stationary targets, typically unmovable, easily found by an enemy, and generally unable to attack enemies preemptively. On the positive side, both can successfully combine a variety of defenses to protect their vital systems and their community. Each line of defense is backed up by another line of defense that kicks in if the previous line of defense fails. Enemies might eventually pierce many lines of defense, but those defenses can discourage or delay attackers long enough for defenders to take additional measures.

IDSs are an important component of multilayer network security. When you deploy an IDS correctly, that IDS can act not only as an early warning system but also as a deterrent. For example, you can configure Snort (which I discuss in the next section) to react to traffic in realtime. Snort's flexible response (flexresp) feature lets you close connections that meet certain criteria. Prevention can be even more important than detection.

Network Intrusion Detection
Network IDSs developed or revised since 1999 operate in hybrid signature-anomaly style—that is, they look both for attack signatures and for protocol anomalies. They watch for matches to canned descriptions of known problems (i.e., signatures and other elements that are consistent enough to look for) as well as for traffic-format and protocol-shape irregularities. Most network IDSs specialize in one or more areas of analysis (e.g., Internet Security System's—ISS's—BlackICE Defender uses protocol-analysis more than signature or pattern matching), but all network IDSs typically have some capability in each area so that they provide overall coverage. Analysis that captures oddities in the traffic is typically faster because packet structure and protocol shape are generally well understood and blueprinted in long-standing Internet Engineering Task Force (IETF) Requests for Comments (RFCs). When a packet with an unlikely structure arrives, the IDS should easily flag the packet as anomalous. Because different network IDSs have different strengths, you need to determine which technology best fits your situation. Let's take a look at how one widely used network IDS—Snort—identifies bad traffic.

The Snort network IDS performs realtime traffic analysis and packet logging on IP networks. This IDS can also perform protocol analysis and content searching and matching, and it can detect a variety of attacks and probes, such as buffer overflows, stealth port scans, Common Gateway Interface (CGI) attacks, Server Message Block (SMB) probes, and OS fingerprinting attempts. Snort uses a flexible rules language that's easy to understand to describe traffic that it should collect or pass and a detection engine with a modular plug-in architecture. Snort's realtime alerting capability incorporates alerting mechanisms for syslog, a user-specified file that stores alerts and logs information about the file system, and for UNIX sockets, which can be used individually or in combination, depending on your needs. In addition, using Samba's smbclient, Snort can send WinPopup messages to Windows clients. You can use Snort as a straight packet sniffer, as a packet logger (e.g., for network traffic debugging), or as a complete network IDS.

Snort is compatible with both Win32 and UNIX systems. The rule base for these systems is the same, which provides excellent cross-platform coordination of rules from one source, and the rules (regardless of type) have a standard structure. In this typical alert rule for a protocol anomaly, which you type on a command line without line breaks

alert ip any any  127.0.0.0/8 any (msg:"BAD TRAFFIC loopback traffic"; classtype: bad-unknown; sid:528; rev:2;)

you can see that the rule specifies an atypical network packet with an address consistent with the loopback address. (This address is a special IP address—127.0.0.1—that's intended as a software loopback interface on a machine; the interface has no hardware associated with it and isn't physically connected to a network.) Finding such a packet on your network is definitely cause for concern because the packet's presence might indicate the use of crafted packets that an intruder has designed for malicious purposes.

Reading the rule from left to right, the Snort sensor (the part of the network IDS that watches network traffic) will fire off an alert if it sees an IP packet (alert ip) coming in or out (any) that matches (<>) the loopback address (127.0.0.0) for any network mask (/8). The message in the Snort log and in any alert window would specify bad traffic, and the Snort classification would be "bad-unknown." The rule classifies how the system reports the transgression (on a scale of 1 through 4, with 1 being most critical), gives it a Snort rule ID (sid) for output sensors (528 falls within the core Snort rule set that ships with Snort), and the revision number of the rule, which is an identifier for tracking purposes. (For information about analyzing packet payload information, see the Web-exclusive sidebar "IDS Signature Analysis," http://www.secadministrator.com, InstantDoc ID 24648.)

Sensors Must See to Believe
For anomalous activity to trigger the sensor (regardless of how the sensor discovers that activity), the sensor must first "see" the traffic. Your network design plays a key role in ensuring that the traffic is directed past the sensor. Ethernet networks come in two basic flavors—switched and nonswitched. Because their communications occur differently, you need to know which type of network you have. (You can find a valuable primer about switched and nonswitched Ethernet networks at http://www.networking.ibm.com/mse/mse0c01.html.) The biggest difference between a switched and a nonswitched Ethernet network is that in a switched environment, the number of collision domains is reduced because packets don't have to go to every machine on the network at the same time. A collision domain is a group of Ethernet or fast Ethernet devices connected by network devices that replicate or regenerate a signal. (For more information about collision domains, see the white paper at http://www.transition.com/learning/whitepapers/downloads/colldom_wp.pdf.)

Nonswitched Ethernet networks. A traditional hub-based Ethernet network is broadcast-based; when one node wants to chat with another node, every machine on the network sees the traffic. You can envision a traditional hub as an open hallway. As you plug twisted-pair cables into the hub, you're opening a door along that hallway for each cable you plug in. You need to assume that a person is in each doorway and that when that person wants to communicate with another person, he or she yells out that person's name, then communicates the message. Although the person responds only when the message is directed to him or her, every person in every doorway hears this communication.

Switched Ethernet networks. Using Switched Ethernet eliminates this behavior (with the exception of broadcast or multicast traffic). Switched Ethernet directs packets from source to destination, which reduces contention. To extend the hallway analogy, the sender writes a message on a piece of paper, folds the paper into an airplane, and sends it directly to the intended recipient. Others in the hallway don't need to deal with this communication.

So what does this communication configuration have to do with your sensors? Understanding how your network is set up and which things can and can't happen on it makes deploying sniffing and sensing equipment easier. Because you need to balance network performance and security, you must be able to take advantage of switched architecture and intrusion detection simultaneously. You need to get all port-directed packets to the place where the sensor can see them.

Placing the Sensor
A sensor can pick up port-directed packets in two places: from the switch backplane and from a Switched Port Analyzer (SPAN) port. A switch's backplane is where the switching process occurs and also where inbound and outbound packets cross before reaching their destination.

The switch backplane. The switch backplane is wider than other I/O ports (i.e., it transmits data at multigigabit speed). Because all the packets must cross over the switch backplane, it's an ideal place to put a network IDS. Several switch manufacturers have plug-in network IDS products and, depending on the hardware you have in place (standalone or cabinet-based), a plugin at this location might be a good option. Alternatively, you can use an external network IDS and the SPAN port.

The SPAN port. A SPAN port is useful on switches because of the differences between hubs and switches that we identified previously. On a hub, when computer A wants to talk to computer B, every port on the hub hears the conversation request, but only computer B answers (regardless of where it's plugged in). On a switch, computer A asks to talk to computer B and the switch delivers the message directly to the port where the media access control (MAC) address for computer B is located (the switch creates MAC tables so that it knows where to send packets intelligently), and no other port hears the request. Because of this direct delivery, if you wanted to monitor the communications, you would need a way for the switch to mirror or copy the traffic between A and B, thus the SPAN port.

When you monitor inbound and outbound traffic to your perimeter firewall and you decide to use a SPAN port, you mirror all traffic to and from that port to your SPAN port, at which you would locate your network IDS. Specifically, you could plug the network IDS directly into the SPAN port with an Ethernet cable. This use of the SPAN port is an accepted practice, but in some instances, you might suffer performance degradations. (For more about monitoring inbound and outbound packets, see the sidebar "Placing Your Controls.")

The first instance of potential degradation is in the mirroring process. Recall that you're asking the switch to make a copy of all the inbound and outbound packets on a particular port (or ports), then forward them to the SPAN port. Second, depending on how you've set up the mirroring, you can overflow the rated transmission speed of the SPAN port. In most cases, the SPAN port is an ordinary 10/100 port; if you try to push more bandwidth through the port than it can handle, you'll saturate the port and it will drop traffic.

Monitoring VLANs
VLANs provide another bit of switching magic that you must plan for when you locate your sensor. (For information about VLANs, go to http://www.sans.org/newlook/resources/IDFAQ/vlan.htm; for a System Administrator, Networking, and Security—SANS—Institute technical brief about VLANs, go to the SANS VLAN FAQ at http://www.intel.com/network/connectivity/resources/doc_library/tech_brief/virtual_lans.htm.) With a VLAN, all the participants on that network might not physically reside on the same wire. VLANs let you break up the ports on a switch, grouping some together to effectively manage your LAN bandwidth. You can arrange your VLANs any way you choose: putting high network users together, separating a test network from a production network, connecting people who perform the same function—even if they reside in different places on the LAN (e.g., they work on different floors). Each VLAN can have its own 100MB or higher network, and high use on one VLAN does not affect the others. However, when you use VLANs, you make the job of network monitoring more difficult because now you can't see all the traffic—only what's in a particular VLAN. You typically need a router to pass traffic from one VLAN to another, and your filtering and monitoring must take place at the head end, where the network traffic coalesces.

Overcoming IDS-Evasion Tactics
After you locate your sensor in the right place to monitor the traffic that interests you, make sure you're running the most recent version of the software or signatures to avoid sensor-evasion mechanisms. Sensor, or network IDS, evasion is an intruder's attempt to circumvent detection by confounding a network IDS or simply hiding attacks from a network IDS within network traffic. For example, an intruder could send significantly fragmented packets to evade older network IDSs. Packets are usually broken apart when datagrams moving between two nodes pass through networks on which the packet size—or Maximum Transmission Unit (MTU)—is smaller than the size of the datagram. Because this "size window" can change as the packet travels, the routers must compute the maximum size of the window at each hop along the route and the packets must be resized accordingly. Packets are fragmented, then reassembled at the destination host in the IP layer. If any fragments are lost, the packets must be retransmitted. The end host knows how to reconstruct the entire datagram by looking at each fragment's fragment ID and offset (from zero) and knows to expect more fragments if the more fragments (MF) bit is set.

Although you expect packets sent through networks to be disassembled and reassembled, an attacker could purposely break data into much smaller sections, aimed at overwhelming the network IDS or simply confusing it. Network IDSs that perform no reassembly are vulnerable to attackers who used fragmented packets to overwhelm them. Fragrouter, for example, is a tool that interferes with the usual datagram sizes to confuse a network IDS. The network IDS has to look at every packet that passes it. Suppose that instead of the usual fragmentation of a 1MB file, I threw one million 1-byte fragments past the IDS. Although each piece is tiny, when all the tiny pieces join other network traffic, older IDSs would be overwhelmed and miss the attack because the IDS performed no reassembly. IDSs created or updated since 1999 perform packet reassembly, making this kind of attack less successful. However, this network IDS evasion technique is just one in a sea of options.

Recently, attackers have used exotic encoding mechanisms to evade IDSs. For example, an attacker can alter HTTP encoding to something nonstandard to completely confound your sensor (i.e., the network IDS doesn't understand the language in which the exploit is encoded). In RFC 2616, the HTTP 1.1 specification has allowances for content encoding that let you compress a document without losing the identity of its underlying media type. (For more information about the ways in which attackers can overthrow network IDS, see Greg Hoglund and Jon Gary, "Multiple Levels of De-synchronization and other concerns with testing an IDS system" at http://www.securityfocus.com/infocus/1204.)

Most mainstream Web servers use either hexadecimal or Unicode Transformation Format (UTF) URL encoding, which modern network IDSs can decode and analyze. However, Microsoft IIS also supports %u encoding, which earlier IDSs weren't aware of and didn't try to decode. (Some but not all current IDSs are aware of this coding.) Attacks that used this brand of encoding could slip by sensors unnoticed. (For information about this vulnerability, see eEye Digital Security's advisory, "%u encoding IDS bypass vulnerability," at http://www.eeye.com/html/research/advisories/ad20010705.html.)

Network IDS Success
Intrusion detection plays an important part in a multilayered defense-in-depth approach. As you deploy a network IDS, you need to know your network's architecture to locate your sensor effectively. You'll find placement a key factor in your network IDS's success in protecting your network from invasion. Of course, I've just scratched the surface of network IDSs.