assembly subroutines get called twice without even being called from main

Question

I'm trying to define some subroutines that have calls to printf in them. A very trivial example is as follows:

extern printf
LINUX        equ     80H
EXIT         equ     60

section .data
    intfmt: db "%ld", 10, 0

segment .text
    global  main

main:
    call os_return      ; return to operating system

os_return:
    mov  rax, EXIT      ; Linux system call 60 i.e. exit ()
    mov  rdi, 0     ; Error code 0 i.e. no errors
    int  LINUX      ; Interrupt Linux kernel

test:
    push rdi
    push rsi
    mov rsi, 10
    mov rdi, intfmt
    xor rax, rax
    call printf
    pop rdi
    pop rsi
    ret

Here test just has a call to printf that outputs the number 10 to the screen. I would not expect this to get called as I have no call to it.

However when compiling and running:

nasm -f elf64 test.asm
gcc -m64 -o test test.o

I get the output:

10
10

I'm totally baffled and wondered if someone could explain why this is happening?

Answer 1

int 80H invokes the 32-bit system call interface, which a) uses the 32-bit system call numbers and b) is intended for use by 32-bit code, not 64-bit code. Your code is actually performing a umask system call with random parameters.

For a 64-bit system call, use the syscall instruction instead:

...
os_return:
    mov  rax, EXIT      ; Linux system call 60 i.e. exit ()
    mov  rdi, 0     ; Error code 0 i.e. no errors
    syscall         ; Interrupt Linux kernel
...

Answer 2

I would say that your call to exit is failing, so when it returns, it falls through to the test function, that prints the first 10.

Then when you return with ret you go back to the instruction just after the call os_return, that is, well os_return. The call to exit fails again and falls through to the test function again. But this time the ret returns from the main function and the program ends.

About why is the exit call failing, I cannot tell as I don't have a 64-bit system available. But you could disassemble the exit function from libc and see how it is done there. My guess is that the int LINUX interface is 32-bit only, as it exists only for historic compatibility, and 64-bit linux in not so old.