How does Linux prioritize custom signal handlers?

Question

We had a lecture last week that involved how the OS (in this case Linux, and in this particular case our school server uses SUSE Linux 11) handles interrupts. One thing of note was that for most signals, you can catch the interrupt and define your own signal handler to run instead of the default. We used an example to illustrate this, and I found what at first seemed to me as interesting behavior. Here's the code:

#include <stdio.h>
#include <signal.h>

#define INPUTLEN 100

main(int ac, char *av[])

{
  void inthandler (int);
  void quithandler (int);
  char input[INPUTLEN];
  int nchars;

  signal(SIGINT, inthandler);
  signal(SIGQUIT, quithandler);

  do {
    printf("\nType a message\n");
    nchars = read(0, input, (INPUTLEN - 1));
    if ( nchars == -1)
      perror("read returned an error");
    else {
      input[nchars] = '\0';
      printf("You typed: %s", input);
    }
  }
  while(strncmp(input, "quit" , 4) != 0); 
}

void inthandler(int s)
{
  printf(" Received Signal %d ....waiting\n", s);
  int i = 0;
  for(int i; i<3; ++i){
    sleep(1);
    printf("inth=%d\n",i);
  }
  printf(" Leaving inthandler \n");
}

void quithandler(int s)
{
  printf(" Received Signal %d ....waiting\n", s);
  for(int i; i<7; ++i){
    sleep(1);
    printf("quith=%d\n",i);
  }  printf(" Leaving quithandler \n");
}

So, when running this code, I expected something like this:

1. Running code.... ^C
2. Enter inthandler, executing loop, hit ^\
3. Exit inthandler, go into quithandler, execute quithandler loop
4. ^C back to inthandler. If I execute ^C again while I'm in the inthandler, ignore successive inthandler signals until the current inthandler is done processing.

I found something that, based on observation, seems like a nested, 2-queue-depth "scheduling" of the signals. If, for example, I enter the following interrupts in quick succession:

• ^C, ^\, ^C, ^\, ^\, ^C

I'll receive the following behavior/output from the code:

^CReceived signal 2 ....waiting
^\Received Signal 3 ....waiting
^C^\^\^C quith=0
quith=1
quith=2
quith=3
quith=4
quith=5
quith=6
quith=7
Leaving quithandler
Received Signal 3 ....waiting
quith=1
quith=2
quith=3
quith=4
quith=5
quith=6
quith=7
Leaving quithandler
inth=0
inth=1
inth=2
inth=3
Leaving inthandler
Received Signal 2 ....waiting
inth=0
inth=1
inth=2
inth=3
Leaving inthandler

In other words, it appears to be processed like this:

1. Receive first ^C signal
2. Receive ^\ signal, "delay" the inthandler and go into quithandler
3. Receive next ^C signal, but because we are "nested" in a inthandler already, put it at the back of the inthandler "queue"
4. Receive quithandler, place at back of quithandler queue.
5. Execute quithandler until queue is empty. Ignore the third quithandler because it seems to only have a queue depth of 2.
6. Leave quithandler, and execute the 2 remaining inthandlers. Ignore the final inthandler because queue-depth of 2.

I showed the behavior to my professor, and he seems to agree that the "nested 2 queue depth" behavior is what's happening, but we're not 100% sure why (he comes from a hardware background and has only just started teaching this class). I wanted to post to SO to see if anybody could shed some light on why/how Linux processes these signals, as we weren't quite expecting some of the behavior i.e. nesting.

I think the test case I wrote out should be enough to illustrate what's going on, but here are a bunch of screenshots of additional test cases:

http://imgur.com/Vya7JeY,fjfmrjd,30YRQfk,uHHXFu5,Pj35NbF

I wanted to leave the additional test cases as a link as they're kind of large screenshots.

Thank you!

Answer 1

The rules (for non-realtime signals, such as the SIGQUIT and SIGINT you're using) are:

1. By default, a signal is masked when its handler is entered, and unmasked when the handler exits;
2. If a signal is raised while it is masked, it is left pending, and will be delivered if/when that signal is unmasked.

The pending state is binary - a signal is either pending or not pending. If a signal is raised multiple times while masked, it will still only be delivered once when unmasked.

So what happens in your example is:

1. SIGINT is raised and the inthandler() signal handler starts executing, with SIGINT masked.
2. SIGQUIT is raised, and the quithandler() signal handler starts executing (interrupting inthandler()) with SIGQUIT masked.
3. SIGINT is raised, adding SIGINT to the set of pending signals (because it is masked).
4. SIGQUIT is raised, adding SIGQUIT to the set of pending signals (because it is masked).
5. SIGQUIT is raised, but nothing happens because SIGQUIT is already pending.
6. SIGINT is raised, but nothing happens because SIGINT is already pending.
7. quithandler() finishes executing, and SIGQUIT is unmasked. Because SIGQUIT is pending, it is then delivered, and quithandler() starts executing again (with SIGQUIT masked again).
8. quithandler() finishes executing for the second time, and SIGQUIT is unmasked. SIGQUIT is not pending, so inthandler() then resumes executing (with SIGINT still masked).
9. inthandler() finishes executing, and SIGINT is unmasked. Because SIGINT is pending, it is then delivered, and inthandler() starts executing again (with SIGINT masked again).
10. inthandler() finishes executing for the second time, and SIGINT is unmasked. The main function then resumes executing.

On Linux, you can see the current set of masked and pending signals for a process by examining /proc/<PID>/status. The masked signals are shown in the SigBlk: bitmask and the pending signals in the SigPnd: bitmask.

If you install your signal handlers with sigaction() rather than signal(), you can specify the SA_NODEFER flag to request that the signal isn't masked while its handler executes. You could try this in your program - with one or both signals - and try to predict what the output will look like.