Inconsistent return values from open function between libc and kernel syscall

Question

When attempting to open a non-existent file on an iOS device, I observed inconsistent return values between the open function from the standard libc library and the kernel syscall (syscall number 5). While the libc open function returns -1, as expected and documented, the kernel syscall open function unexpectedly returns 2. This discrepancy necessitates clarification and documentation to understand the underlying behavior and ensure predictable results when working with file I/O operations in iOS.

Code snippet for ASM code:

#if defined(__arm64__)

#define __asm_syscall(...) do { \
asm volatile ("svc 0x80" : "=r"(x0) : __VA_ARGS__ : "memory", "cc"); \
return x0; \
} while (0)

__attribute__((always_inline))
static inline long asm_open(const void* __path, int __flags, int __mode) {
    register long x16 __asm__("x16") = 5; // 5: open
    register long x0 __asm__("x0") = (long)__path;
    register long x1 __asm__("x1") = (long)__flags;
    register long x2 __asm__("x2") = (long)__mode;
    __asm_syscall("r"(x16), "0"(x0), "r"(x1), "r"(x2));
}

#endif

This is how I call the function

    char file_path[1024];
    int fd = 0;

    // Set the file path
    strcpy(file_path, getenv("HOME"));
    strcat(file_path, "/Documents/non-existent.txt");
    
    // Open file
    fd = (int)asm_open(file_path, (O_RDWR | O_CREAT), 0666); // -> This returns 2 instead of -1 like standard open function from libc
    LOGI("[INFO] : Open %d", fd); // [INFO] : Open 2

Answer 1

On the machine code level, the system call either returns either a file descriptor and a clear carry flag or an error code and the carry flag set. The libc shim translates this into the usual convention with setting errno and returning -1 on error.

Other system calls may have even larger divergences between machine code level and C level. I recommend going through the libc wrapper if at all possible.