close() x86_64 system call weird return value

Question

My xinetd daemon suddenly stopped working after a kernel upgrade (from 2.6.24 to 2.6.33). I've run an strace and found this:

[...]
close(3)                                = 0
munmap(0x7f1a93b43000, 4096)            = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=8*1024, rlim_max=16*1024}) = 0
setrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
close(3)                                = 4294967287
exit_group(1)                           = ?

So basically, it looks like the close system call returned something different than 0 or -1

I did several tests and it appears that it happens only with 64bit executables:

$ file closetest32
closetest32: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped
$ strace closetest32
execve("./closetest32", ["closetest32"], [/* 286 vars */]) = 0
[ Process PID=4731 runs in 32 bit mode. ]
open("/proc/mounts", O_RDONLY)          = 3
close(3)                                = 0
close(3)                                = -1 EBADF (Bad file descriptor)
_exit(0)                                = ?


$ file closetest64
closetest64: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), statically linked, not stripped
$ strace closetest64
execve("./closetest64", ["closetest64"], [/* 286 vars */]) = 0
open("/proc/mounts", O_RDONLY)          = 3
close(3)                                = 0
close(3)                                = 4294967287
_exit(0)                                = ?

I'm running the following kernel:

Linux foobar01 2.6.33.9-rt31.64.el5rt #1 SMP PREEMPT RT Wed May 4 10:34:12 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

The worst part is that I cannot reproduce the bug on another machine with the same kernel.

Any ideas ?

EDIT: as requested: here's the code used for closetest32 and closetest64

closetest32.asm:

.section .data

filename:
    .ascii "/proc/mounts"

.section .text
.globl _start
_start:
    xorl %edi, %edi
    movl $5, %eax # open() i386 system call
    leal filename, %ebx # %ebx ---> filename
    movl $0, %esi # O_RDONLY flag into esi
    int $0x80

    xorl %edi, %edi
    movl $6, %eax # close() i386 system call
    movl $3, %ebx # fd 3
    int $0x80

    xorl %edi, %edi
    movl $6, %eax # close() i386 system call
    movl $3, %ebx # fd 3
    int $0x80

    ## terminate program via _exit () system call
    movl $1, %eax # %eax =  _exit() i386 system call
    xorl %ebx, %ebx # %ebx = 0 normal program return code
    int $0x80

compiled as:

as test32.asm -o test32.o --32
ld -m elf_i386 test32.o -o closetest32

closetest64.asm:

.section .data

filename:
    .ascii "/proc/mounts"

.section .text
.globl _start
_start:
    xorq %rdi, %rdi
    movq $2, %rax # open() system call
    leaq filename, %rdi # %rdi ---> filename
    movq $0, %rsi # O_RDONLY flag into rsi
    syscall

    xorq %rdi, %rdi
    movq $3, %rax # close() system call
    movq $3, %rdi # fd 3
    syscall

    xorq %rdi, %rdi
    movq $3, %rax # close() system call
    movq $3, %rdi # fd 3
    syscall

    ## terminate program via _exit () system call
    movq $60, %rax # %rax = _exit() system call
    xorq %rdi, %rdi # %rdi = 0 normal program return code
    syscall

compilation:

as test64.asm -o test64.o
ld test64.o -o closetest64

Answer 1

As expected, a rollback to a previous kernel version solved the problem. I'm not really a kernel specialist but as far as I understand, the answer given by @R.. makes sense:

This is a 64-bit machine, so 1<<32-9 should never appear. The problem is that the kernel is internally using unsigned instead of int for the return value of some of these functions, then returning -EBADF which gets reduced modulo 2^32 rather than modulo 2^64

The problem is that the generic code in the libc syscall wrappers that handles syscall error returns has to treat the return value as a long (since it could be a pointer or long for some syscalls) when making the comparison to see if it's a small negative value that would indicate an error. But the kernel returned (long)(unsigned)-9 which is very different from (long)-9. or (unsigned long)-9 (either of which would have worked).