FreeBSD syscall clobbering more registers than Linux? Inline asm different behaviour between optimization levels

Question

Recently I was playing with freebsd system calls I had no problem for i386 part since its well documented at here. But i can't find same document for x86_64.

I saw people are using same way like on linux but they use just assembly not c. I suppose in my case system call actually changing some register which is used by high optimization level so it gives different behaviour.

/* for SYS_* constants */
#include <sys/syscall.h>

/* for types like size_t */
#include <unistd.h>

ssize_t sys_write(int fd, const void *data, size_t size){
    register long res __asm__("rax");
    register long arg0 __asm__("rdi") = fd;
    register long arg1 __asm__("rsi") = (long)data;
    register long arg2 __asm__("rdx") = size;
    __asm__ __volatile__(
        "syscall"
        : "=r" (res)
        : "0" (SYS_write), "r" (arg0), "r" (arg1), "r" (arg2)
        : "rcx", "r11", "memory"
    );
    return res;
}

int main(){
    for(int i = 0; i < 1000; i++){
        char a = 0;
        int some_invalid_fd = -1;
        sys_write(some_invalid_fd, &a, 1);
    }
    return 0;
}

In above code I just expect it to call sys_write 1000 times then return main. I use truss to check system call and their parameters. Everything works fine with -O0 but when I go -O3 for loop getting stuck forever. I believe system call changing i variable or 1000 to something weird.

Dump of assembler code for function main:

0x0000000000201900 <+0>:     push   %rbp
0x0000000000201901 <+1>:     mov    %rsp,%rbp
0x0000000000201904 <+4>:     mov    $0x3e8,%r8d
0x000000000020190a <+10>:    lea    -0x1(%rbp),%rsi
0x000000000020190e <+14>:    mov    $0x1,%edx
0x0000000000201913 <+19>:    mov    $0xffffffffffffffff,%rdi
0x000000000020191a <+26>:    nopw   0x0(%rax,%rax,1)
0x0000000000201920 <+32>:    movb   $0x0,-0x1(%rbp)
0x0000000000201924 <+36>:    mov    $0x4,%eax
0x0000000000201929 <+41>:    syscall 
0x000000000020192b <+43>:    add    $0xffffffff,%r8d
0x000000000020192f <+47>:    jne    0x201920 <main+32>
0x0000000000201931 <+49>:    xor    %eax,%eax
0x0000000000201933 <+51>:    pop    %rbp
0x0000000000201934 <+52>:    ret

What is wrong with sys_write()? Why for loop getting stuck?

Answer 1

Optimization level determines where clang decides to keep its loop counter: in memory (unoptimized) or in a register, in this case r8d (optimized). R8D is a logical choice for the compiler: it's a call-clobbered reg it can use without saving at the start/end of main, and you've told it all the registers it could use without a REX prefix (like ECX) are either inputs / outputs or clobbers for the asm statement.

Note: if FreeBSD is like MacOS, system call error / no-error status is returned in CF (the carry flag), not via RAX being in the -4095..-1 range. In that case, you'd want a GCC6 flag-output operand like "=@ccc" (err) for int err(#ifdef __GCC_ASM_FLAG_OUTPUTS__ - example) or a setc %cl in the template to materialize a boolean manually. (CL is a good choice because you can just use it as an output instead of a clobber.)

---

FreeBSD's syscall handling trashes R8, R9, and R10, in addition to the bare minimum clobbering the Linux does: RAX (retval) and RCX / R11 (The syscall instruction itself uses them to save RIP / RFLAGS so the kernel can find its way back to user-space, so the kernel never even sees the original values.)

Possibly also RDX, we're not sure; the comments call it "return value 2" (i.e. as part of a RDX:RAX return value?). We also don't know what future-proof ABI guarantees FreeBSD intends to maintain in future kernels.

You can't assume R8-R10 are zero after syscall because they're actually preserved instead of zeroed when tracing / single-stepping. (Because then the kernel chooses not to return via sysret, for the same reason as Linux: hardware / design bugs make that unsafe if registers might have been modified by ptrace while inside the system call. e.g. attempting to sysret with a non-canonical RIP will #GP in ring 0 (kernel mode) on Intel CPUs! That's a disaster because RSP = user stack at that point.)

---

The relevant kernel code is the sysret path (well spotted by @NateEldredge; I found the syscall entry point by searching for swapgs, but hadn't gotten to looking at the return path).

The function-call-preserved registers don't need to be restored by that code because calling a C function didn't destroy them in the first place. and the code does restore the function-call-clobbered "legacy" registers RDI, RSI, and RDX.

R8-R11 are the registers that are call-clobbered in the function-calling convention, and that are outside the original 8 x86 registers. So that's what makes them "special". (R11 doesn't get zeroed; syscall/sysret uses it for RFLAGS, so that's the value you'll find there after syscall)

Zeroing is slightly faster than loading them, and in the normal case (syscall instruction inside a libc wrapper function) you're about to return to a caller that's only assuming the function-calling convention, and thus will assume that R8-R11 are trashed (same for RDI, RSI, RDX, and RCX, although FreeBSD does bother to restore those for some reason.)

---

This zeroing only happens when not single-stepping or tracing (e.g. truss or GDB si). The syscall entry point into an amd64 kernel (Github) does save all the incoming registers, so they're available to be restored by other ways out of the kernel.

---

Updated asm() wrapper

// Should be fixed for FreeBSD, plus other improvements
ssize_t sys_write(int fd, const void *data, size_t size){
    register ssize_t res __asm__("rax");
    register int arg0 __asm__("edi") = fd;
    register const void *arg1 __asm__("rsi") = data;  // you can use real types
    register size_t arg2 __asm__("rdx") = size;
    __asm__ __volatile__(
        "syscall"
                    // RDX *maybe* clobbered
        : "=a" (res), "+r" (arg2)
                           // RDI, RSI preserved
        : "a" (SYS_write), "r" (arg0), "r" (arg1)
          // An arg in R10, R8, or R9 definitely would be
        : "rcx", "r11", "memory", "r8", "r9", "r10"   ////// The fix: r8-r10
         // see below for a version that avoids the "memory" clobber with a dummy input operand
    );
    return res;
}

Use "+r" output/input operands with any args that need register long arg3 asm("r10") or similar for r8 or r9.

This is inside a wrapper function so the modified value of the C variables get thrown away, forcing repeated calls to set up the args every time. That would be the "defensive" approach until another answer identifies more definitely-non-trashed registers.

---

I did break *0x000000000020192b then info registers when break happened. r8 is zero. Program still gets stuck in this case

I assume that r8 wasn't zero before you did that GDB continue across the syscall instruction. Yes, that test confirms that the FreeBSD kernel is trashing r8 when not single-stepping. (And behaving in a way that matches what we see in the source code.)

---

Note that you can tell the compiler that a write system call only reads memory (not writes) using a dummy "m" input operand instead of a "memory" clobber. That would let it hoist the store of c out of the loop. (How can I indicate that the memory *pointed* to by an inline ASM argument may be used?)

i.e. "m"(*(const char (*)[size]) data) as an input instead of a "memory" clobber.

If you're going to write specific wrappers for each syscall you use, instead of a generic wrapper you use for every 3-operand syscall that just casts all operands to unsigned long, this is the advantage you can get from doing that.

Speaking of which, there's absolutely no point in making your syscall args all be long; making user-space sign-extend int fd into a 64-bit register is just wasted instructions. The kernel ABI will (almost certainly) ignore the high bytes of registers for narrow args, like Linux does. (Again, unless you're making a generic syscall3 wrapper that you just use with different SYS_ numbers to define write, read, and other 3-operand system calls; then you would cast everything to register-width and just use a "memory" clobber).

I made these changes for my modified version below.

Also note that for RDI, RSI, and RDX, there are specific-register letter constraints which you can use instead of register-asm locals, just like you're doing for the return value in RAX ("=a"). BTW, you don't really need a matching constraint for the call number, just use an "a" input; it's easier to read because you don't need to look at another operand to check that you're matching the right output.

// assuming RDX *is* clobbered.
// could remove the + if it isn't.
ssize_t sys_write(int fd, const void *data, size_t size)
{
    // register long arg3 __asm__("r10") = ??;
    // register-asm is useful for R8 and up

    ssize_t res;
    __asm__ __volatile__("syscall"
                    // RDX
        : "=a" (res), "+d" (size)
         //  EAX/RAX       RDI       RSI
        : "a" (SYS_write), "D" (fd), "S" (data),
          "m" (*(const char (*)[size]) data) // tells compiler this mem is an input
        : "rcx", "r11"    //, "memory"
#ifndef __linux__
              , "r8", "r9", "r10"   // Linux always restores these
#endif
    );
    return res;
}

Some people prefer register ... asm("") for all the operands because you get to use the full register name, and don't have to remember the totally-non-obvious "D" for RDI/EDI/DI/DIL vs. "d" for RDX/EDX/DX/DL

Answer 2

Here's a test framework to work with. It is [loosely] modeled on a H/W logic analyzer and/or things like dtrace.

It will save registers before and after the syscall instruction in a large global buffer.

After the loop terminates it will dump out a trace of all the register values that were stored.

---

It is multiple files. To extract:

1. save the code below to a file (e.g. /tmp/archive).
2. Create a directory: (e.g.) /tmp/extract
3. cd to /tmp/extract.
4. Then do: perl /tmp/archive -go.
5. It will create some subdirectories: /tmp/extract/syscall and /tmp/extract/snaplib and store a few files there.
6. cd to the program target directory (e.g.) cd /tmp/extract/syscall
7. build with: make
8. Then, run with: ./syscall

---

Here is the file:

Edit: I've added a check for overflow of the snaplist buffer in the snapnow function. If the buffer is full, dumpall is called automatically. This is good in general but also necessary if the loop in main never terminates (i.e. without the check the post loop dump would never occur)

Edit: And, I've added optional "x86_64 red zone" support

#!/usr/bin/perl
# FILE: ovcbin/ovcext.pm 755
# ovcbin/ovcext.pm -- ovrcat archive extractor
#
# this is a self extracting archive
# after the __DATA__ line, files are separated by:
#   % filename

ovcext_cmd(@ARGV);
exit(0);

sub ovcext_cmd
{
    my(@argv) = @_;
    local($xfdata);
    local($xfdiv,$divcur,%ovcdiv_lookup);

    $pgmtail = "ovcext";
    ovcinit();
    ovcopt(\@argv,qw(opt_go opt_f opt_t));

    $xfdata = "ovrcat::DATA";
    $xfdata = \*$xfdata;

    ovceval($xfdata);

    ovcfifo($zipflg_all);

    ovcline($xfdata);

    $code = ovcwait();

    ovcclose(\$xfdata);

    ovcdiv();

    ovczipd_spl()
        if ($zipflg_spl);
}

sub ovceval
{
    my($xfdata) = @_;
    my($buf,$err);

    {
        $buf = <$xfdata>;
        chomp($buf);

        last unless ($buf =~ s/^%\s+([\@\$;])/$1/);

        eval($buf);

        $err = $@;
        unless ($err) {
            undef($buf);
            last;
        }

        chomp($err);
        $err = " (" . $err . ")"
    }

    sysfault("ovceval: bad options line -- '%s'%s\n",$buf,$err)
        if (defined($buf));
}

sub ovcline
{
    my($xfdata) = @_;
    my($buf);
    my($tail);

    while ($buf = <$xfdata>) {
        chomp($buf);

        if ($buf =~ /^%\s+(.+)$/) {
            $tail = $1;
            ovcdiv($tail);
            next;
        }

        print($xfdiv $buf,"\n")
            if (ref($xfdiv));
    }

}

sub ovcdiv
{
    my($ofile) = @_;
    my($mode);
    my($xfcur);
    my($err,$prt);

    ($ofile,$mode) = split(" ",$ofile);

    $mode = oct($mode);
    $mode &= 0777;

    {
        unless (defined($ofile)) {
            while ((undef,$divcur) = each(%ovcdiv_lookup)) {
                close($divcur->{div_xfdst});
            }
            last;
        }

        $ofile = ovctail($ofile);

        $divcur = $ovcdiv_lookup{$ofile};
        if (ref($divcur)) {
            $xfdiv = $divcur->{div_xfdst};
            last;
        }
        undef($xfdiv);

        if (-e $ofile) {
            msg("ovcdiv: file '%s' already exists -- ",$ofile);

            unless ($opt_f) {
                msg("rerun with -f to force\n");
                last;
            }

            msg("overwriting!\n");
        }

        unless (defined($err)) {
            ovcmkdir($1)
                if ($ofile =~ m,^(.+)/[^/]+$,);
        }

        msg("$pgmtail: %s %s",ovcnogo("extracting"),$ofile);
        msg(" chmod %3.3o",$mode)
            if ($mode);
        msg("\n");

        last unless ($opt_go);
        last if (defined($err));

        $xfcur = ovcopen(">$ofile");

        $divcur = {};
        $ovcdiv_lookup{$ofile} = $divcur;

        if ($mode) {
            chmod($mode,$xfcur);
            $divcur->{div_mode} = $mode;
        }

        $divcur->{div_xfdst} = $xfcur;
        $xfdiv = $xfcur;
    }
}

sub ovcinit
{

    {
        last if (defined($ztmp));
        $ztmp = "/tmp/ovrcat_zip";

        $PWD = $ENV{PWD};

        $quo_2 = '"';

        $ztmp_inp = $ztmp . "_0";
        $ztmp_out = $ztmp . "_1";
        $ztmp_perl = $ztmp . "_perl";

        ovcunlink();

        $ovcdbg = ($ENV{"ZPXHOWOVC"} != 0);
    }
}

sub ovcunlink
{

    _ovcunlink($ztmp_inp,1);
    _ovcunlink($ztmp_out,1);
    _ovcunlink($ztmp_perl,($pgmtail ne "ovcext") || $opt_go);
}

sub _ovcunlink
{
    my($file,$rmflg) = @_;
    my($found,$tag);

    {
        last unless (defined($file));

        $found = (-e $file);

        $tag //= "notfound"
            unless ($found);
        $tag //= $rmflg ? "cleaning" : "keeping";

        msg("ovcunlink: %s %s ...\n",$tag,$file)
            if (($found or $ovcdbg) and (! $ovcunlink_quiet));

        unlink($file)
            if ($rmflg and $found);
    }
}

sub ovcopt
{
    my($argv) = @_;
    my($opt);

    while (1) {
        $opt = $argv->[0];
        last unless ($opt =~ s/^-/opt_/);

        shift(@$argv);
        $$opt = 1;
    }
}

sub ovctail
{
    my($file,$sub) = @_;
    my(@file);

    $file =~ s,^/,,;
    @file = split("/",$file);

    $sub //= 2;

    @file = splice(@file,-$sub)
        if (@file >= $sub);

    $file = join("/",@file);

    $file;
}

sub ovcmkdir
{
    my($odir) = @_;
    my(@lhs,@rhs);

    @rhs = split("/",$odir);

    foreach $rhs (@rhs) {
        push(@lhs,$rhs);

        $odir = join("/",@lhs);

        if ($opt_go) {
            next if (-d $odir);
        }
        else {
            next if ($ovcmkdir{$odir});
            $ovcmkdir{$odir} = 1;
        }

        msg("$pgmtail: %s %s ...\n",ovcnogo("mkdir"),$odir);

        next unless ($opt_go);

        mkdir($odir) or
            sysfault("$pgmtail: unable to mkdir '%s' -- $!\n",$odir);
    }
}

sub ovcopen
{
    my($file,$who) = @_;
    my($xf);

    $who //= $pgmtail;
    $who //= "ovcopen";

    open($xf,$file) or
        sysfault("$who: unable to open '%s' -- $!\n",$file);

    $xf;
}

sub ovcclose
{
    my($xfp) = @_;
    my($ref);
    my($xf);

    {
        $ref = ref($xfp);
        last unless ($ref);

        if ($ref eq "GLOB") {
            close($xfp);
            last;
        }

        if ($ref eq "REF") {
            $xf = $$xfp;
            if (ref($xf) eq "GLOB") {
                close($xf);
                undef($$xfp);
            }
        }
    }

    undef($xf);

    $xf;
}

sub ovcnogo
{
    my($str) = @_;

    unless ($opt_go) {
        $str = "NOGO-$str";
        $nogo_msg = 1;
    }

    $str;
}

sub ovcdbg
{

    if ($ovcdbg) {
        printf(STDERR @_);
    }
}

sub msg
{

    printf(STDERR @_);
}

sub msgv
{

    $_ = join(" ",@_);
    print(STDERR $_,"\n");
}

sub sysfault
{

    printf(STDERR @_);
    exit(1);
}

sub ovcfifo
{
}

sub ovcwait
{
    my($code);

    if ($pid_fifo) {
        waitpid($pid_fifo,0);
        $code = $? >> 8;
    }

    $code;
}

sub prtstr
{
    my($val,$fmtpos,$fmtneg) = @_;

    {
        unless (defined($val)) {
            $val = "undef";
            last;
        }

        if (ref($val)) {
            $val = sprintf("(%s)",$val);
            last;
        }

        $fmtpos //= "'%s'";

        if (defined($fmtneg) && ($val <= 0)) {
            $val = sprintf($fmtneg,$val);
            last;
        }

        $val = sprintf($fmtpos,$val);
    }

    $val;
}

sub prtnum
{
    my($val) = @_;

    $val = prtstr($val,"%d");

    $val;
}

END {
    msg("$pgmtail: rerun with -go to actually do it\n")
        if ($nogo_msg);
    ovcunlink();
}

1;
package ovrcat;
__DATA__
% ;
% syscall/syscall.c
/* for SYS_* constants */
#include <sys/syscall.h>

/* for types like size_t */
#include <unistd.h>

#include <snaplib/snaplib.h>

ssize_t
my_write(int fd, const void *data, size_t size)
{
    register long res __asm__("rax");
    register long arg0 __asm__("rdi") = fd;
    register long arg1 __asm__("rsi") = (long)data;
    register long arg2 __asm__("rdx") = size;

    __asm__ __volatile__(
        SNAPNOW
        "\tsyscall\n"
        SNAPNOW
        : "=r" (res)
        : "0" (SYS_write), "r" (arg0), "r" (arg1), "r" (arg2)
        : "rcx", "r11", "memory"
    );

    return res;
}

int
main(void)
{

    for (int i = 0; i < 8000; i++) {
        char a = 0;
        int some_invalid_fd = -1;
        my_write(some_invalid_fd, &a, 1);
    }

    snapreg_dumpall();

    return 0;
}
% snaplib/snaplib.h
// snaplib/snaplib.h -- register save/dump

#ifndef _snaplib_snaplib_h_
#define _snaplib_snaplib_h_

#ifdef _SNAPLIB_GLO_
#define EXTRN_SNAPLIB       /**/
#else
#define EXTRN_SNAPLIB       extern
#endif

#ifdef RED_ZONE
#define SNAPNOW \
    "\tsubq\t$128,%%rsp\n" \
    "\tcall\tsnapreg\n" \
    "\taddq\t$128,%%rsp\n"
#else
#define SNAPNOW     "\tcall\tsnapreg\n"
#endif

typedef unsigned long reg_t;

#ifndef SNAPREG
#define SNAPREG     (1500 * 2)
#endif

typedef struct {
    reg_t snap_regs[16];
} __attribute__((packed)) snapreg_t;
typedef snapreg_t *snapreg_p;

EXTRN_SNAPLIB snapreg_t snaplist[SNAPREG];

#ifdef _SNAPLIB_GLO_
snapreg_p snapcur = &snaplist[0];
snapreg_p snapend = &snaplist[SNAPREG];
#else
extern snapreg_p snapcur;
extern snapreg_p snapend;
#endif

#include <snaplib/snaplib.proto>

#include <snaplib/snapgen.h>

#endif
% snaplib/snapall.c
// snaplib/snapall.c -- dump routines

#define _SNAPLIB_GLO_
#include <snaplib/snaplib.h>
#include <stdio.h>
#include <stdlib.h>

void
snapreg_dumpall(void)
{
    snapreg_p cur = snaplist;
    snapreg_p endp = (snapreg_p) snapcur;

    int idx = 0;
    for (;  cur < endp;  ++cur, ++idx) {
        printf("\n");
        printf("%d:\n",idx);
        snapreg_dumpgen(cur);
    }

    snapcur = snaplist;
}

// snapreg_crash -- invoke dump and abort
void
snapreg_crash(void)
{

    snapreg_dumpall();
    exit(9);
}

// snapreg_dumpone -- dump single element
void
snapreg_dumpone(snapreg_p cur,int regidx,const char *regname)
{
    reg_t regval = cur->snap_regs[regidx];

    printf("  %3s %16.16lX %ld\n",regname,regval,regval);
}
% snaplib/snapreg.s
    .text
    .globl  snapreg
snapreg:
    push    %r14
    push    %r15
    movq    snapcur(%rip),%r15
    movq    %rax,0(%r15)
    movq    %rbx,8(%r15)
    movq    %rcx,16(%r15)
    movq    %rdx,24(%r15)
    movq    %rsi,32(%r15)
    movq    %rsi,40(%r15)
    movq    %rbp,48(%r15)
    movq    %rsp,56(%r15)
    movq    %r8,64(%r15)
    movq    %r9,72(%r15)
    movq    %r10,80(%r15)
    movq    %r11,88(%r15)
    movq    %r12,96(%r15)
    movq    %r13,104(%r15)
    movq    %r14,112(%r15)
    movq    0(%rsp),%r14
    movq    %r14,120(%r15)
    addq    $128,%r15
    movq    %r15,snapcur(%rip)
    cmpq    snapend(%rip),%r15
    jae     snapreg_crash
    pop %r15
    pop %r14
    ret
% snaplib/snapgen.h
#ifndef _snapreg_snapgen_h_
#define _snapreg_snapgen_h_
static inline void
snapreg_dumpgen(snapreg_p cur)
{
    snapreg_dumpone(cur,0,"rax");
    snapreg_dumpone(cur,1,"rbx");
    snapreg_dumpone(cur,2,"rcx");
    snapreg_dumpone(cur,3,"rdx");
    snapreg_dumpone(cur,5,"rsi");
    snapreg_dumpone(cur,5,"rsi");
    snapreg_dumpone(cur,6,"rbp");
    snapreg_dumpone(cur,7,"rsp");
    snapreg_dumpone(cur,8,"r8");
    snapreg_dumpone(cur,9,"r9");
    snapreg_dumpone(cur,10,"r10");
    snapreg_dumpone(cur,11,"r11");
    snapreg_dumpone(cur,12,"r12");
    snapreg_dumpone(cur,13,"r13");
    snapreg_dumpone(cur,14,"r14");
    snapreg_dumpone(cur,15,"r15");
}
#endif
% snaplib/snaplib.proto
// /home/cae/OBJ/ovrgen/snaplib/snaplib.proto -- prototypes

// FILE: /home/cae/preserve/ovrbnc/snaplib/snapall.c
// snaplib/snapall.c -- dump routines

    void
    snapreg_dumpall(void);

    // snapreg_crash -- invoke dump and abort
    void
    snapreg_crash(void);

    // snapreg_dumpone -- dump single element
    void
    snapreg_dumpone(snapreg_p cur,int regidx,const char *regname);
% syscall/Makefile
# /home/cae/preserve/ovrbnc/syscall -- makefile
PGMTGT += syscall
LIBSRC += ../snaplib/snapreg.s
LIBSRC += ../snaplib/snapall.c
ifndef COPTS
    COPTS += -O2
endif
CFLAGS += $(COPTS)
CFLAGS += -mno-red-zone
CFLAGS += -g
CFLAGS += -Wall
CFLAGS += -Werror
CFLAGS += -I..
all: $(PGMTGT)
syscall: syscall.c $(CURSRC) $(LIBSRC)
    cc -o syscall $(CFLAGS) syscall.c $(CURSRC) $(LIBSRC)
clean:
    rm -f $(PGMTGT)