How to figure out function prototype from assembly code?

Question

I have some assembly code and I want to find out the calling function prototype, so that I can all the function from c++ code.

What I am really trying to do is to inject a dll into a running process and call the running process's functions form my dll. Right now I have successfully injected my dll but don't know how to do the 'calling'.

I am newbie and I only know a little about assembly code. My dll is written in visual c++ 2012.

Here is the code of the running process:

CPU Disasm
Address   Hex dump          Command                                  Comments
6013BE24  /$  53            PUSH EBX
6013BE25  |.  8B1D 10461860 MOV EBX,DWORD PTR DS:[60184610]
6013BE2B  |.  8B1B          MOV EBX,DWORD PTR DS:[EBX]
6013BE2D  |.  8B40 04       MOV EAX,DWORD PTR DS:[EAX+4]
6013BE30  |.  FFD3          CALL EBX
6013BE32  |.  5B            POP EBX
6013BE33  \.  C3            RETN

You can see at 6013BE30 it is calling the function, and the function(EBX) is at 004BAFAC

CPU Disasm
Address   Hex dump          Command                                  Comments
004BAFAC  /$  55            PUSH EBP                                 ; Test.004BAFAC(guessed void)
004BAFAD  |.  8BEC          MOV EBP,ESP
004BAFAF  |.  53            PUSH EBX
004BAFB0  |.  8BD8          MOV EBX,EAX
004BAFB2  |.  8B43 08       MOV EAX,DWORD PTR DS:[EBX+8]
004BAFB5  |.  E8 3EF40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFBA  |.  8B43 0C       MOV EAX,DWORD PTR DS:[EBX+0C]
004BAFBD  |.  E8 36F40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFC2  |.  8B43 14       MOV EAX,DWORD PTR DS:[EBX+14]
004BAFC5  |.  E8 2EF40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFCA  |.  8B43 18       MOV EAX,DWORD PTR DS:[EBX+18]
004BAFCD  |.  E8 26F40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFD2  |.  8B43 1C       MOV EAX,DWORD PTR DS:[EBX+1C]
004BAFD5  |.  E8 1EF40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFDA  |.  8B43 10       MOV EAX,DWORD PTR DS:[EBX+10]
004BAFDD  |.  E8 16F40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFE2  |.  8B43 20       MOV EAX,DWORD PTR DS:[EBX+20]
004BAFE5  |.  E8 0EF40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFEA  |.  8B43 44       MOV EAX,DWORD PTR DS:[EBX+44]
004BAFED  |.  E8 06F40B00   CALL 0057A3F8                            ; [Test.0057A3F8
004BAFF2  |.  8B43 44       MOV EAX,DWORD PTR DS:[EBX+44]

So how to call the function at 004BAFAC from visual c++?

Answer 1

In assembly function arguments are popped from the stack, starting with the last one. So to pass arguments to a function one pushes them to the stack first, then call the function. In MASM this would look something like that for showing a messagebox:

.data 
MsgBoxCaption  db "Attention",0 
MsgBoxText     db "Hello Message Box!",0 

.code
start:
push 0
mov eax, offset MsgBoxCaption
push eax
push offset MsgBoxText
push 0
call MessageBoxA
call ExitProcess
end start

Whilst in C++ it would look something like that:

int retval = MessageBox(NULL, "Hello Message Box!", "Attention", 0);

Disassembled with OllyDbg you'd get:

CPU Disasm
Address   Hex dump          Command                                  Comments
011D13C0  |.  6A 00         PUSH 0                                   ; /Type = MB_OK|MB_DEFBUTTON1|MB_APPLMODAL
011D13C2  |.  68 54571D01   PUSH OFFSET 011D5754                     ; |Caption = "Attention"
011D13C7  |.  68 3C571D01   PUSH OFFSET 011D573C                     ; |Text = "Hello Message Box!"
011D13CC  |.  6A 00         PUSH 0                                   ; |hOwner = NULL
011D13CE  |.  FF15 40831D01 CALL DWORD PTR DS:[<&USER32.MessageBoxA> ; \USER32.MessageBoxA

The return value gets stored in eax. You see that in case of datatypes bigger than a DWORD (4 bytes) you are not actually passing values but references. Each argument is actually just a DWORD. That is because the stack is 32 bit aligned in a 32 bit system, and therefore can only hold 32 bit values. The same is the case for the registers. In a 32 bit system registers are 32 bit wide. If you pass a string you are passing a reference to the string, not the string nor a variable width array of characters/bytes. In case of an Integer it will most likely be passed as value, not reference. When prototyping functions in MASM prototyping all parameters as DWORD will work (not in C++ tho).

So in your example above it seems like there is one parameter getting passed onto the stack (push ebx). I say seems like because it could very well just be that ebx is pushed to store it for later retrieval (i.e. after the call ebx is popped again - maybe the value stored in ebx needed to be preserved). You can only tell for sure when actually running a live debugging session and observing the stack/registers. Step over the call and check your stack pointer (esp) before and after to see how many arguments are passed to the function (again: 1 argument = 1 DWORD = 4 bytes).

To prototype your function in C++ you will need to figure out what datatype the argument is, possibly by single-stepping through the assembly code and checking the registers and/or the stack while traversing the code around the function in question. The same holds true for the return value in eax. So eax will never hold a string, but rather possibly a reference to a string, but it might hold a 32 bit or smaller integer. You'd have to find out what kind of data eax points to by reversing the surrounding code as well.

EDIT: Actually half the stuff I wrote was wrong, most notably parameters always being references, so please reread what I have written. Sorry for the inconvenience - shouldnt answer questions being tired :)

Answer 2

Without more detail, it's really difficult to tell what's going on. Due to the unusual code, I suspect that the app has been obfuscated. You might have better luck with IDA.

The short answer is that all the functions seem to take a pointer-to-something in eax (0057A3F8 might take a 4-byte integer instead). It's difficult to tell what these are pointers to. You will probably need an assembly helper to call these from C++.

---

First you need to work out the calling convention: Both 6013BE24 and 004BAFAC appear to take a single argument in eax, which according to this list on Wikipedia is only used by Borland (fastcall), Watcom (by default), or Embarcadero Delphi. You haven't said if the app you're reverse-engineering is written in C++, but even if it is, I think you'll have difficulty calling without a little assembly.

6013BE24 appears look up a function in a function table with something like ebx=**(void***)0x60184610 (I forget what pointer-to-pointer-to-function-pointer looks like); I'd see if this pointed to a relocation table somewhere. It also takes a pointer to a struct in eax (something like struct Foo { void * dummy; Bar * bar; ... }) and then does roughly return ebx(eax->bar). I don't know if it actually returns anything, but it will return anything returned in registers. I don't know why it uses ebx instead of a register that doesn't need to be preserved.

004BAFAC sets up a stack frame, preserves ebx, and then does makes a few calls to 0057A3F8 with things in bar (i.e. eax). If bar is a void**, it'd look something like f(bar[2]); f(bar[3]); f(bar[5]); f(bar[6]); f(bar[7]); f(bar[4]); f(bar[8]); f(bar[17]); and then it loads bar[17] again (and there the disassembly ends). I suspect bar is actually a struct of some sort.

0057A3F8 appears to take a single argument in eax.