Building an Incident Response/Audit CD
Whether you are working as a security incident responder or a system auditor you should have a response or analysis CD in your tool kit. While there are many different bootable systems available, what if you want to do response without rebooting? What if the tools that you need aren't on the CD? What if there is no CD available for your operating system? This article covers a step by step process for creating a security response/analysis CD of your own that's customized for your system.
In both our compliance auditing and incident response/forensics practice we make heavy use of customized CDs full of analysis tools. Let’s take a look at the process of building one step by step. For our example we’re going to use Linux but this process really works for any UNIX based system you have.
The first thing that you need to do is to create a directory that you'll use to hold all of your security tools while building your CD. Generally it's easier if you build a directory structure that's familiar, so we'll recommend that you first create a top level folder that will essentially be the root of the security tools CD that you create. For our purposes we'll call this directory "response".
Inside of the "response" folder you will want to create an organized directory structure. In reality you can just dump everything into the root of the security CD but it's much better to keep things organized. Depending on the task I may have any number of folders here; for instance, I may include a "scripts" folder, a "forensic_tools" folder, etc. At a minimum, however, I will always have a "bin" and "lib" directory for housing both the security tools and the dynamic libraries that they require.
Create the directories for your security CD:
mkdir response
cd response
mkdir bin
mkdir lib
This is probably the hardest step in our process. What you need to do is determine exactly which security or analysis tools you want to include on your CD. Personally, it makes a lot of sense to me to create one complete security CD with everything that I think I'll ever need. Here are some basic suggestions just to get us going:
bash, csh, etc. - If there's a shell you prefer to use, make sure you've got a copy of it on your CD. If we're not going to trust the tool binaries on the CD why would we trust the shell?
ls, cd, cat, more, etc. - If we're doing incident response or forensics we clearly can't trust anything on the system that we're analyzing. This means that even the basics must be brought along with us on our CD.
lsof - If I had to pick just one tool to bring along for incident response or security analysis of a system it would be lsof. Since UNIX systems view pretty much everything as a file, the ability to view open files becomes incredibly important. This tool is worth an entire article all by itself!
netstat, ifconfig, ifstatus, tcpdump, etc. - Basic networking tools are a must. If you think that you might be dealing with a kernel level rootkit these tools will still be fooled, but for a userspace rootkit these tools are invaluable for figuring out what's happening at the network level.
netcat, dd, icat, ils, fls, etc. - With this list we start getting into really specialized functionality. Netcat likely needs no introduction. Netcat has been termed the "Swiss Army Knife" of network tools. If I need to get data on or off of a box quickly this is almost always my tool of choice. Added to that we include a variety of tools from The Sleuthkit. More often than not when doing incident response I've found it very useful to be able to examine the contents of a particular inode that has obviously been deleted. Certainly you could image the drive and extract it offline but sometimes there's no substitute for a fast answer!
Now the list above is by no means intended to be a complete list. As mentioned already, we always include our own tools and scripts. What do you do with them now, though? Take your list of tools and copy each binary into the "bin" directory in your tools directory:
Copy everything you want in your toolbox into bin:
cp /bin/ls bin
cp /bin/netstat bin
...
So far we're in good shape. Our next step isn't hard but it's the one that usually prevents people from building their own UNIX response or security CD. UNIX binaries are, by default, compiled to use dynamic libraries. This allows not only for good reuse of code and smaller disk sizes, it also leads to smaller in-memory sizes since the shared libraries are only loaded one time rather than being duplicated throughout the memory space of the system. This means that we either need to recompile every tool that we want to put on our CD as a static binary (unpleasant at best) or somehow bring along the dynamic libraries.
As it turns out, there's a really easy to use tool that can tell you which libraries any UNIX binary needs: ldd. The ldd tool (List Dynamic Dependencies) will tell you which version of the library is necessary and where on the system that library currently resides. What we need to do is to extract this dynamic library information from all of the security tools that we have in our "bin" folder and copy them into the "lib" folder. Here's a little UNIX magic to make this happen: (To use this, your working directory should be "response")
Resolve and copy library dependencies
ldd bin/* | grep "/lib" | sed -e 's/.*\(\/lib\/[^ ]*\).*/cp \1 lib/' | sh
What that command run you should now have every required library in your "lib" folder for every command in your "bin" folder!
Our next task is optional. At this point you really could just copy this whole bundle into an archive, copy it to your Windows host and unpack it onto a CD that you burn from there. When dealing with UNIX tools, though, I find that it's often better to actually make the ISO image on the UNIX system too so that you have control over what the file attributes will be, especially the execute bit for your security tools.
This might sound like a hard task but there's actually a great file system tool available on Linux systems in particular but which has been ported to all major UNIX systems: mkisofs. "mkisofs" allows you to point it at a directory structure and it will then convert that into an ISO that can be used to burn a CD! To do this, if you've been following along, just do the following:
Use "mkisofs" to turn "response" into an ISO:
cd ..
mkisofs -o ResponseCD.iso response
Your ISO is actually a disk image, so you can now take this to any platform and use your favorite CD burning tool to burn this image out as a CD. How do you use the CD now that you have it?
To use the CD, you take it to the system that you want to examine and mount it. Once it's mounted you need to set your PATH variable to point to wherever your "response/bin" directory is on the mounted CD and the LD_LIBRARY_PATH to wherever the "response/lib" directory is. Now execute a clean shell (bash, csh, etc.) from the CD and you're ready to go!!!
This is just one of the many topics discussed and taught hands on in David Hoelzer’s class, “Advanced System & Network Auditing”, available through The SANS Institute.
Step One:
The first thing that you need to do is to create a directory that you'll use to hold all of your security tools while building your CD. Generally it's easier if you build a directory structure that's familiar, so we'll recommend that you first create a top level folder that will essentially be the root of the security tools CD that you create. For our purposes we'll call this directory "response".
Inside of the "response" folder you will want to create an organized directory structure. In reality you can just dump everything into the root of the security CD but it's much better to keep things organized. Depending on the task I may have any number of folders here; for instance, I may include a "scripts" folder, a "forensic_tools" folder, etc. At a minimum, however, I will always have a "bin" and "lib" directory for housing both the security tools and the dynamic libraries that they require.
Create the directories for your security CD:
mkdir response
cd response
mkdir bin
mkdir lib
Step Two:
This is probably the hardest step in our process. What you need to do is determine exactly which security or analysis tools you want to include on your CD. Personally, it makes a lot of sense to me to create one complete security CD with everything that I think I'll ever need. Here are some basic suggestions just to get us going:
bash, csh, etc. - If there's a shell you prefer to use, make sure you've got a copy of it on your CD. If we're not going to trust the tool binaries on the CD why would we trust the shell?
ls, cd, cat, more, etc. - If we're doing incident response or forensics we clearly can't trust anything on the system that we're analyzing. This means that even the basics must be brought along with us on our CD.
lsof - If I had to pick just one tool to bring along for incident response or security analysis of a system it would be lsof. Since UNIX systems view pretty much everything as a file, the ability to view open files becomes incredibly important. This tool is worth an entire article all by itself!
netstat, ifconfig, ifstatus, tcpdump, etc. - Basic networking tools are a must. If you think that you might be dealing with a kernel level rootkit these tools will still be fooled, but for a userspace rootkit these tools are invaluable for figuring out what's happening at the network level.
netcat, dd, icat, ils, fls, etc. - With this list we start getting into really specialized functionality. Netcat likely needs no introduction. Netcat has been termed the "Swiss Army Knife" of network tools. If I need to get data on or off of a box quickly this is almost always my tool of choice. Added to that we include a variety of tools from The Sleuthkit. More often than not when doing incident response I've found it very useful to be able to examine the contents of a particular inode that has obviously been deleted. Certainly you could image the drive and extract it offline but sometimes there's no substitute for a fast answer!
Now the list above is by no means intended to be a complete list. As mentioned already, we always include our own tools and scripts. What do you do with them now, though? Take your list of tools and copy each binary into the "bin" directory in your tools directory:
Copy everything you want in your toolbox into bin:
cp /bin/ls bin
cp /bin/netstat bin
...
Step Three:
So far we're in good shape. Our next step isn't hard but it's the one that usually prevents people from building their own UNIX response or security CD. UNIX binaries are, by default, compiled to use dynamic libraries. This allows not only for good reuse of code and smaller disk sizes, it also leads to smaller in-memory sizes since the shared libraries are only loaded one time rather than being duplicated throughout the memory space of the system. This means that we either need to recompile every tool that we want to put on our CD as a static binary (unpleasant at best) or somehow bring along the dynamic libraries.
As it turns out, there's a really easy to use tool that can tell you which libraries any UNIX binary needs: ldd. The ldd tool (List Dynamic Dependencies) will tell you which version of the library is necessary and where on the system that library currently resides. What we need to do is to extract this dynamic library information from all of the security tools that we have in our "bin" folder and copy them into the "lib" folder. Here's a little UNIX magic to make this happen: (To use this, your working directory should be "response")
Resolve and copy library dependencies
ldd bin/* | grep "/lib" | sed -e 's/.*\(\/lib\/[^ ]*\).*/cp \1 lib/' | sh
What that command run you should now have every required library in your "lib" folder for every command in your "bin" folder!
Step Four: (Optional)
Our next task is optional. At this point you really could just copy this whole bundle into an archive, copy it to your Windows host and unpack it onto a CD that you burn from there. When dealing with UNIX tools, though, I find that it's often better to actually make the ISO image on the UNIX system too so that you have control over what the file attributes will be, especially the execute bit for your security tools.
This might sound like a hard task but there's actually a great file system tool available on Linux systems in particular but which has been ported to all major UNIX systems: mkisofs. "mkisofs" allows you to point it at a directory structure and it will then convert that into an ISO that can be used to burn a CD! To do this, if you've been following along, just do the following:
Use "mkisofs" to turn "response" into an ISO:
cd ..
mkisofs -o ResponseCD.iso response
Your ISO is actually a disk image, so you can now take this to any platform and use your favorite CD burning tool to burn this image out as a CD. How do you use the CD now that you have it?
To use the CD, you take it to the system that you want to examine and mount it. Once it's mounted you need to set your PATH variable to point to wherever your "response/bin" directory is on the mounted CD and the LD_LIBRARY_PATH to wherever the "response/lib" directory is. Now execute a clean shell (bash, csh, etc.) from the CD and you're ready to go!!!
This is just one of the many topics discussed and taught hands on in David Hoelzer’s class, “Advanced System & Network Auditing”, available through The SANS Institute.