Unix O_CREAT flag without mode specified

Question

The definition of the UNIX open() function when used with the O_CREAT flag is that it requires a third argument named mode in order to set the files' privileges.

What if that mode is not specified?

int file;
static const char filename[] = "test.test";

if ((file = open(filename, O_RDWR | O_CREAT | O_TRUNC)) == 1)
{
    perror("Error opening file.");
    exit(EXIT_FAILURE);
}

close(file);

What happens with the file that is created using those flags? On my system I get:

-r--r-s---  1 hyperboreean hyperboreean     0 2009-02-25 01:40 test.test

A theory is that the open function looks on the stack and checks for the mode parameter and ends up using a random integer it finds.

What does the standard say about this?

Answer 1

The POSIX standard (IEEE 1003.1:2008) prototypes open() as:

int open(const char *path, int oflag, ...);

The section describing the behaviour of O_CREAT doesn't say what will happen if you omit the necessary third argument, which means the behaviour is undefined - anything is possible.

In practice, the use of part of the stack that was intended to be stack frame or return address or something similar is quite likely - unto a reasonable approximation, that can be considered a random integer.

The POSIX 2008 standard has some interesting new (and useful) flags for open(), including:

• O_FDCLOEXEC to specify close-on-exec at open.
• O_DIRECTORY to specify that the file must be a directory.
• O_NOFOLLOW to specify not to chase symlinks.

Answer 2

Good question. The mode value will be modified by the umask of the process. So if you don't pass a mode explicitly to open in an O_CREAT operation, and if this results in random bits being used for the mode, those random bits will be modified by the umask.

Wish I could be more definitive and precise, but I agree with cdonner that "random" values are being used, along with the umask.

Edit: One thing you could try is to use dtruss or truss or some other facility to trace system calls, and look at the value of mode at run-time to see if something sensible is used, or if it's just random bits modified by the umask, for example.

Answer 3

hyperboreean, your suspicion may be not so far off the mark. I was hoping to find the answer in Kernighan Ritchie. Unfortunately, I did not. I think the permissions parameter is required with the O_CREAT flag, and if you don't provide it, open() will pull a random value from the stack, which of course goes unnoticed in C.

Edit: by "random" I mean not predictable. It probably is picking up part of the return address, which sits on top of the parameters on the stack.