Take a look at the free, open source Metasploit penetration testing framework www.metasploit.com and you'll see that literally hundreds of the vulnerabilities that it can exploit have the same root cause: a buffer overflow. These types of vulnerabilities can occur on just about any platform, including Windows, Linux and UNIX, and when exploited can lead to the complete compromise of the machine in question.
So what are buffer overflows, exactly, and why are they so common?
In the most simple terms, a buffer overflow happens when an application receives a bigger chunk of data than it is expecting, with the result that the data doesn't fit into the allocated storage space, or buffer. A common example might be an application which asks for a username it expects to be no longer than, say, 8 characters. If the user enters a username of more than 8 characters there is a potential problem if the application tries to store the username in a string buffer of 8 bytes, which can take a maximum of 8 letters.
So the big question is what happens to the extra characters? What may happen is that they "overflow" the 8 byte buffer, and get written into the adjacent buffers, if any have been defined, overwriting any data they contain.
That in itself is bad, because any data previously held in those buffers is now corrupted. But things go from bad to worse if the data that overflows is actually program code written by a malicious hacker. If the hacker can find a way to get that code to execute he can quite possibly get complete remote control of the computer.
The reason that buffer overflow errors are so common is that human programmers make errors: they fail to foresee that unexpected data values may be entered, and they fail to carry out bounds checking to ensure that any data that a user enters falls within the expected range. Many application development languages carry out automatic bounds checking , but unfortunately this is not the case with two very popular languages, C and C++, which are used to develop many applications.
One way to get overflow data to execute is to use a string long enough to overwrite multiple memory segments until the string data overflows into the saved Instruction Pointer register, EIP. (EIP controls the execution flow of the application by storing the address of the next instruction to be executed.) Then it's a matter of making sure that the data that overflows into EIP is (or points to) the address of the start of the malicious program code that has overflowed elsewhere.
Exploiting buffer overflows sounds easy in theory, but how easy is it to exploit a known buffer overflow error in practice? The first thing you would need to do is get hold of the application which has the buffer overflow error so that you could get to work crafting an exploit. In order to see what is going on inside the computer when the application crashes, you would need to attach the application to a debugger program. The debugger of choice for many Windows hackers is Ollydbg (www.ollydbg.de), because it is free, easy to use, and provides a clear view of registers such as EIP.
To get started you would need to crash the application a number of times by entering simple strings (perhaps consisting of nothing but As) of varying lengths and then use the debugger to provide you with a view of what is happening when the system crashes, including what happens to the contents of the EIP register. If you're lucky you'll see EIP get overwritten with your letter As. Since EIP is a 32 bit register, and can therefore hold 4 characters, and since the ASCII code for A is hex 41, Ollydbg would show EIP now has a value of 41414141. If that's the case then you would know that that your long username was overflowing into EIP.
The next step would be to find out exactly which four As overflow into it. One way to do this is to enter a username string of, say, 1000 As followed by 1000 Bs. If As show up in EIP again, then you would know that the four characters that overflow into EIP are in the first half of the string. Then you would enter a username string of 500 As and 1500 Bs, and repeat the process, continuing to do this until you could identify that, for example, the 304th 305th, 306th and 307th characters in your username string were the ones which will end up in EIP. You could now set any arbitrary value you choose into EIP (by placing them as the 304th 305th, 306th and 307th characters in your username string,) and therefore you would now have control over the execution flow of the application.
The next step would be to set EIP to point to a memory address into which some other part of your username string has overflowed. If the data that overflows to this address is your malicious code, the system would then execute your code and do whatever you wanted - which would probably be to provide a remote shell to that machine so that you have complete control over it. Malicious code that does this is called shellcode.
If you look in Ollydbg at the value of some of the other hardware registers, you may find following a crash that the Stack Pointer register, ESP, points to an address which has been overwritten with overflow data. If that's the case then that could be an ideal place to place your shellcode. Using the binary tree procedure used earlier to find which characters end up in EIP, you can also work out which ones will overflow into the address that ESP points to.
The next step would then be to get hold of the shellcode required to give you control of the machine. This is very easy - you would simply generate this code using the Metasploit framework's msfpayload function.
After this, you would need to a way to set EIP to point to the address that ESP points to, which is where your shellcode will be located. Ideally you would use the command JMP ESP to jump to the address that ESP points to, but EIP is a register which holds addresses, not commands. The solution is to set EIP to point to the address of a JMP ESP command already loaded in memory, which will then redirect the execution flow to your shellcode.
Ollydbg can help you do that, by letting you take a closer look at some core system dlls like USER32.dll. Within this dll you'll find the jmp ESP command at a particular memory address, so putting this address into EIP will make it execute the JMP ESP command, which sets the execution flow to resume at the address pointed to in ESP - which is your shellcode!
Finally, you would need to build your username string which contains your shellcode and the address of the JMP ESP command that you want to overflow into EIP, located at the right parts of the string to overflow to the right destinations.
Although slightly simplified, that's all it would take to exploit a suitable buffer overflow error in an application to gain remote control of a machine running the application - unless hardware or software measures are in place to protect against this.
Protection against buffer overflow errors
A stack cookie, or "canary," is essentially a randomized piece of data that an application can be made (using a compiler option)to write to the stack just before EIP. That means that if data overflows from its assigned buffer into EIP, it will overwrite the stack cookie too. Just as canaries where used in mines in the 19th century to indicate whether the air was healthy or toxic, the application uses the state of the stack cookie canary to check the health of the system. If the stack cookie has been changed then this indicates a buffer overflow, and the application is terminated.
Data Execution Protection
Hardware DEP, which is available with specific processors, basically marks certain areas of the stack as non-executable areas. That means that any shellcode places in the stack simply won't execute.
Address Space Layout Randomization (ASLR)
ASLR, a feature of many modern operating systems, randomizes the base address of executables, dlls and other items in a process's address space. These addresses change with each system boot, making it impossible to use JMP commands in system dlls, since their locations are no longer predictable.
Although these technologies can make exploiting a buffer overflow considerably harder, all three of them can be overcome in certain circumstances.