What are some conditions that may cause fork() or system() calls to fail on Linux?

Question

And how can one find out whether any of them are occuring, and leading to an error returned by fork() or system()? In other words, if fork() or system() returns with an error, what are some things in Linux that I can check to diagnose why that particular error is happening?

For example:

• Just plain out of memory (results in errno ENOMEM) - check memory use with 'free' etc.
• Not enough memory for kernel to copy page tables and other accounting information of parent process (results in errno EAGAIN)
• Is there a global process limit? (results in errno EAGAIN also?)
• Is there a per-user process limit? How can I find out what it is?
• ...?

Answer 1

And how can one find out whether any of them are occuring?

Check the errno value if the result (return value) is -1

From the man page on Linux:

RETURN VALUE
On success, the PID of the child process is returned in the parent, and 0 is returned in the child. On failure, -1 is returned in the parent, no child process is created, and errno is set appropriately.

ERRORS
EAGAIN
fork() cannot allocate sufficient memory to copy the parent's page tables and allocate a task structure for the child.
EAGAIN
It was not possible to create a new process because the caller's RLIMIT_NPROC resource limit was encountered. To exceed this limit, the process must have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability.
ENOMEM
fork() failed to allocate the necessary kernel structures because memory is tight.

CONFORMING TO SVr4, 4.3BSD, POSIX.1-2001.

Answer 2

nproc in /etc/security/limits.conf can limit the number of processes per user.

You can check for failure by examining the return from fork. A 0 means you are in the child, a positive number is the pid of the child and means you are in the parent, and a negative number means the fork failed. When fork fails it sets the external variable errno. You can use the functions in errno.h to examine it. I normally just use perror to print the error (with some text prepended to it) to stderr.

#include <stdio.h>
#include <errno.h>
#include <unistd.h>

int main(int argc, char** argv) {
    pid_t pid;

    pid = fork();
    if (pid == -1) {
        perror("Could not fork: ");
        return 1;
    } else if (pid == 0) {
        printf("in child\n");
        return 0;
    };

    printf("in parent, child is %d\n", pid);

    return 0;
}