ASM call conventions

Question

I have been reading about calling conventions in ASM and this is what I got so far:

          x86(userland)    x86(kernel)    x64(userland)    x64(kernel)

1st arg           Stack           EBX               RDI            RDI
2nd arg           Stack           ECX               RSI            RSI
3rd arg           Stack           EDX               RDX            RDX
4th arg           Stack           ESI               RCX            R10
5th arg           Stack           EDI               R8             R8
6th arg           Stack           EBP               R9             R9

result            EAX             EAX               RAX            RAX

My questions are:

1. Is what I have learned so far correct?

2. How can I pass more than 6 arguments in x86(kernel) and x64(both)? Using the stack? Mind to show me a small example?

3. I have a kernel module and I'm willing to call a function in that module from ASM. What convention should I use? kernel or userland?

Answer 1

1) Yes, it seems to but only for Linux. I think you can rely on the Linux conventions described here: http://www.x86-64.org/documentation/abi.pdf. But actually you are free to pass arguments the way it is described in the intel assembly manual chapter 6.3.3

2) Using the stack is the way the compiler does it:

int func(int i, int j, int k, int l, int m, int n, int o, int p, int q) { return q; }
void func2() { func(1, 2, 3, 4, 5, 6, 7, 8, 9); }

Then:

$ gcc -c func.c && objdump -d func.o 

Which outputs on my x86_64 machine:

0000000000000000 <func>:
   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   89 7d fc                mov    %edi,-0x4(%rbp)
   7:   89 75 f8                mov    %esi,-0x8(%rbp)
   a:   89 55 f4                mov    %edx,-0xc(%rbp)
   d:   89 4d f0                mov    %ecx,-0x10(%rbp)
  10:   44 89 45 ec             mov    %r8d,-0x14(%rbp)
  14:   44 89 4d e8             mov    %r9d,-0x18(%rbp)
  18:   8b 45 20                mov    0x20(%rbp),%eax
  1b:   5d                      pop    %rbp
  1c:   c3                      retq   

000000000000001d <func2>:
  1d:   55                      push   %rbp
  1e:   48 89 e5                mov    %rsp,%rbp
  21:   48 83 ec 18             sub    $0x18,%rsp
  25:   c7 44 24 10 09 00 00    movl   $0x9,0x10(%rsp)
  2c:   00 
  2d:   c7 44 24 08 08 00 00    movl   $0x8,0x8(%rsp)
  34:   00 
  35:   c7 04 24 07 00 00 00    movl   $0x7,(%rsp)
  3c:   41 b9 06 00 00 00       mov    $0x6,%r9d
  42:   41 b8 05 00 00 00       mov    $0x5,%r8d
  48:   b9 04 00 00 00          mov    $0x4,%ecx
  4d:   ba 03 00 00 00          mov    $0x3,%edx
  52:   be 02 00 00 00          mov    $0x2,%esi
  57:   bf 01 00 00 00          mov    $0x1,%edi
  5c:   e8 00 00 00 00          callq  61 <func2+0x44>
  61:   c9                      leaveq 
  62:   c3                      retq   

3) I would say kernel since you're calling the function inside a kernel module. To have a full valid example, you could call your function from C in your module and disassembly the .ko the same way I did to see how the compiler handles it. It should be straight forward.

Answer 2

I only code for x86, and can give you some feedback for that architecture (the first two columns).

As to 3., if it's a kernel function (as opposed to, say, a libc function), you would use the kernel conventions (your column 2).

As to 1., correct, except that you wouldn't use ebx for the 6th argument. The traditional function prologue would push this argument assuming it were the actual ebp. So the cutoff is actually 5 arguments.

As to 2., if you have more than 5 arguments, you would store them consecutively in memory, and hand on a pointer to the beginning of this memory region in ebx.