Considering getting into reverse engineering/disassembly

Question

Assuming a decent understanding of assembly on common CPU architectures (eg: x86), how can one explore a potential path (career, fun and profit, etc) into the field of reverse engineering? There is so little educational guides out there so it is difficult to understand what potential uses this has today (eg: is searching for buffer overflow exploits still common, or do stack monitoring programs make this obselete?). I am not looking for any step by step program, just some relevant information such as tips on how to efficiently find a specific area of a program. Basic things in the trade. As well as what it is currently being used for today.

So to recap, what current uses does reverse engineering yield today? And how can one find some basic information on how to learn the trade (again it doesn't have to be step-by-step, just anything which can through a clue would be helpful).

Answer 1

The main one that I know of, as mentioned previously, is malware related. One of the primary tasks of researchers that work for scanner companies is to take a sample and debug it in a lab or virtual environment.

Along those same lines, there are plenty of security related areas that use reverse engineering/disassembly. Computer forensics is an area you might want to look into. A confiscated computer might have contain command and control programs (but no source) for various activities (command and control botnet, DoS attack programs, etc). It is usually much easier to circumvent protected data schemes by reverse engineering the program that protects rather than figuring out the password or key.

DRM/security protection in both hardware and software is a big reverse engineering area. Note this could be on either "side" of the issue (and law). Consider DVD copy programs, protection removal, the ability to play iTunes music on other devices, the ability to run homebrew programs on Wii, parallelizing a PS3 grid, unlocking an iPhone, etc, etc. Obviously there are many nonlegal only options too (reverse engineer a slot machine timer, ATM machine authentication, etc).

Legacy program conversion is a huge opportunity in many areas, especially government, finance, manufacturing, etc. There are mission critical programs that have run for 30 years on an ancient mainframe or mini that no one has source code to. Teams have to reverse engineer the program to convert it to something newer.

The other suggestions about learning Win32 tutorials are great. Also, sadly, some of the best published work is going to be on cracking (games). Search on that and there are a few tutorials out there which show the basics. A class I took used the book "The Art of Computer Virus Research and Defense" by Peter Szor but it was more heavy on the malware ideas and not the exact disassembly part.

Depending on which route you take you will need a background in other things but knowing assembly is going to be your most critical skill. Not just from an "I understand what that code does mostly" standpoint - you should be able to write stuff from scratch and understand exactly what a given bit of code does and understand other ways that same code could be written. Assembly (coding) involves figuring out one solution to a problem and coding it. Disassembly involves figuring out which of many many diverse solutions was used initially to solve the problem - MUCH harder ;)

Answer 2

I've read that reverse-engineering is used in the security field to understand the internal working of malware and trojans (not too sure about viruses). For articles on reverse engineering as used in the security field, check out www.openrce.org.

Also reverse-engineering doesn't always involve disassembly. For applications written in languages like Java or C#, decompilers usually yield more information about the code than disassemblers.

As my personal interest is in Win32 reverse-engineering, I can only explain my opinions for this particular OS. Can't help you in Linux reverse-engineering then :(

I find the freeware version of IDA Pro 4.9 an excellent disassembler. It detects system libraries so that you don't spend time mucking around the wrong places :) Coupled with a debugger like OllyDbg, you're ready to tackle most any reversing projects for Win32.

If you go down the Win32 route, you'll eventually need to understand the PE structure, maybe unpacking and stuff, but the key thing now is to understand x86 assembly. The disassembled code for Win32 apps is relatively easy to understand if you've done Win32 API coding in languages like C.

To better understand 32 bit assembly, either disassemble your own apps and see how your source code correspond to the disassembly output or learn how to code Win32 API apps with assembly language via Iczelion's Win32 Assembly tutorials.