Linux - Why doesn't a custom system call work properly with negative numbers?

Question

I wrote a custom system call that compares two integers and returns the biggest one. Here's my kernel-side code:

max.c

#include <linux/kernel.h>
#include <linux/syscalls.h>

asmlinkage long sys_max(int num1, int num2) 
{
  if (num1 > num2) 
  {
    return num1;
  }

  else 
  {
    return num2;
  }
}

And here's my user-space code:

max.h

#include <unistd.h>
#define SYS_MAX 323

int max(int num1, int num2)
{
  int maxnumber = syscall(SYS_MAX, num1, num2);
  return maxnumber;
}

I'm using this little program to test the system call:

#include <stdio.h>
#include <max.h>

int main()
{
    int a, b;
    scanf("%d", &a);
    scanf("%d", &b);
    printf("%d", max(a, b));
    return 0;
}

It's working great for positive numbers, or when one is positive and the other negative, but max always returns -1 when dealing with two negative values. I've wondered if this is because of the int->long conversion, but I can't seem to understand what's causing the problem.

Answer 1

If a system call returns a negative value it's treated as an error, and special error handling code is invoked in libc. Namely, the return value is negated and moved into the errno global variable, and the return value of the system call becomes -1.

Actually, while reading about this I discovered that on Linux, only values from -4095 to -1 are treated as errors, but this is neither portable, nor helpful if you want your function to work on any possible values.

In short, you can't safely use the return value of a syscall to return a value that might be negative. The usual convention would be to pass a pointer to a destination variable to hold the result, and reserve the return code for success/failure. Note that when doing this with a syscall you will be working with a user-space pointer from kernel-space, so copy_to_user will be necessary to write the result.