Buffer-Overflow Attacks Explained

Over the past few years, buffer-overflow attacks have become a major security threat to both the Windows and UNIX segments of the IT world. The System Administration, Networking, and Security (SANS) Institute lists buffer-overflow attacks in its article "The Twenty Most Critical Internet Security Vulnerabilities" (http://www.sans.org/top20.htm).

Buffer-overflow attacks are nothing new. The first buffer-overflow attack that infected thousands of Internet-connected machines was the infamous Internet worm released in 1988. That attack exploited a buffer overflow in the finger program and used the overflow to gain access to VAX machines that were running BSD UNIX. The first buffer-overflow vulnerability that hit Microsoft software (Microsoft Internet Information Server—IIS—4.0) was the Malformed HTR Request vulnerability that eEye Digital Security discovered in 1999. To find out more this vulnerability and how to protect against it, see the Microsoft Security Bulletin MS99-019 ("Patch Available for 'Malformed HRT Request' Vulnerability") at http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/bulletin/ms99-019.asp. Although Web servers such as IIS are a favorite buffer-overflow attack target, intruders can launch attacks against any type of application.

The origin of buffer-overflow attacks lies in the C programming language. Like any program, a C program runs in computer memory. In memory, the computer's OS shuffles a program's data and instructions between different memory locations in the address space C programmers use the concept of an array to buffer (i.e., store) program data in computer memory. An array occupies a contiguous set of memory locations and is designed to hold a finite amount of data. The problem with most C compilers (the software engines that execute the C code) is that they don't perform "boundary checking." In other words, malicious code might be allowed to overflow buffer boundaries and overwrite the original code or variables. Intruders can use buffer overflows not only to inject hostile code into memory but also to generate Denial of Service (DoS) conditions on a particular process (e.g., a Web server process). The injected hostile code can instruct a computer to delete or change data—or even disclose confidential information. For a detailed technical explanation of how buffer-overflow attacks work, read "Exploiting Windows NT 4 Buffer Overruns (A Case Study: RASMAN.EXE)," available from the @stake Web site (http://www.atstake.com/research/reports/wprasbuf.html).

Although the C language might be at the root of buffer-overflow attacks, poor programming and administration practices facilitate the attacks. Far too often, systems execute code in a security context that has too many system privileges. You certainly find this problem in the Windows world. Administrators and programmers must honor the principle of least privilege, giving programs only the privileges needed to perform their function. C programmers must limit the use of dangerous functions in their code. Functions such as strcpy() make it relatively easy to copy strings to other memory locations and overflow buffer boundaries. Programmers might also want to limit the use of large monolithic pieces of C code: Large chunks of code are more difficult to manage, and they create a bigger risk of security holes.