Assembly language for Reverse Engineering [closed]

Question

What should I choose NASM or MASM for learning assembly. I want to learn assembly, motivation being Reverse Engineering.

So that when I disassemble some executable, I can understand the code by looking at disassembled code.

Update: I think I dint make my self clear.. I understand those are assemblers, but to understand the output of a disassembler I need to know assembly and that's the reason I'm asking where to start(with MASM or NASM)

Answer 1

Assuming you want to learn how to do reverse-engineering on Windows, here's how. The Linux Way is very different (although eventually you get to more or less the same place):

You want to start with something very simple, like 8080 processor. You can find an online emulator here: http://www.tramm.li/i8080 -- this is a legacy OS called "CP/M". Click on "Start EMU8080" link -- you will get a command line of "SID" -- a simple ancient debugger. It only has like 10 commands -- to browse memory at any address, to enter new assembly commands etc. Type HELP to see them all. You can try and google for a SID manual if you want. Then google for 8080 assembly manual -- you will find a bunch of PDFs -- they will all be copies of the same 2 manuals written by Intel in 1970s. Read both -- they will give you a good start into 808x CPU basics. All modern processors used in PCs are derived from 8080. Play with SID and 8080 until you feel like you got the idea.

Next step is 8086. For this you'll need to get a copy of Window XP and run DEBUG program in command line. DEBUG is Microsoft's rip-off of SID. It will be very easy to learn once you know SID. The documentation is here: http://thestarman.pcministry.com/asm/debug/debug.htm -- read as much as you can, also make sure to follow the links to other pages -- the same site contains bunch of stuff on basic 8086 programming. If by that time you will have internalized the 8080 manuals, 8086 will feel super-easy. (If you skip the 8080/SID phase, everything will feel weird and the puzzle will take more time to sort-out, trust me.)

You may also want to play with http://ladsoft.tripod.com/grdb_debugger.html -- an improved clone of DEBUG, easier to work with.

Next step, get a book by Peter Abel called "IBM PC Assembler Language and Programming" -- the older edition, the better. You can find one on Amazon for like 1 buck (I'm not kidding!). Read first 7 chapters. Get yourself assembler called "FASM" (forget MASM and NASM crap). Read FASM documentation. Write a program in 16 bit real mode. You can still run them on Windows XP command line.

Next step, read http://www.drpaulcarter.com/pcasm/ -- this will help you understand protected mode. Then read the actual Intel 386 reference documentation: http://www.logix.cz/michal/doc/i386/

Then buy a book called "Reversing. Secrets of Reverse Engineering". It's pretty lame, but will give you some ideas on what you need to learn next. Like low-level Windows programming (messages, handlers etc.)

Once you got through all this, only then can you download http://www.ollydbg.de/ and try playing with it. Be sure to get version 1.x, not 2.x. Go to Youtube and find some tutorials. You are welcome to skip over to this step, but I can guarantee you will get frustrated and won't understand anything you see on the screen unless you follow through all of the above.

Good luck. If you have any questions you can contact me via my blog (the link is on my profile page).

Answer 2

You're talking about disassembly here. The tools you've both listed are assemblers, which won't help you understand or make sense of existing code. They'll be useful for the re-assembly phase, however!

For disassembly, I highly recommend IDA Pro. It has all of the tools, documentation, plugins, and community help that you seek. As a bonus, the older versions are also free.