Can't provoke Priority Inversion in C++

Question

I'm trying to provoke Priority Inversion on a small C++ program for demonstration purposes but I can't: The low priority thread that holds the mutex is not preempted and keeps running on the critical section. This is what I'm doing:

// let's declare a global mutex
pthread_mutex_t my_mutex;
  ...

int main(int argc, char **argv) {
  ...
  pthread_t normal_thread;
  pthread_t prio_thread;

  pthread_mutexattr_t attr;
  pthread_mutexattr_init (&attr);
  pthread_mutexattr_setprotocol (&attr, PTHREAD_PRIO_NONE);  // ! None !
  pthread_mutex_init(&my_mutex, &attr);

  // create first normal thread (L):
  pthread_create(&normal_thread, NULL, the_locking_start_routine, NULL);

  // just to help the normal thread enter in the critical section
  sleep(2);

  // now will launch:
  // * (M) several CPU intensive SCHED_FIFO threads with priority < 99
  // * (H) one SCHED_FIFO thread that will try to lock the mutex, with priority < 99

  // build Real Time attributes for the Real Time threads:
  pthread_attr_t my_rt_att;
  pthread_attr_init(&my_rt_att);

  // it was missing in the original post and it was also wrong:
  // even setting the SchedPolicy you have to set "InheritSched"
  pthread_attr_setinheritsched(&my_rt_att, PTHREAD_EXPLICIT_SCHED)

  pthread_attr_setschedpolicy(&my_rt_att, SCHED_FIFO);
  struct sched_param params;

  params.sched_priority = 1;
  pthread_attr_setschedparam(&my_rt_att, &params);

  pthread_create(&prio_thread, &my_rt_att, the_CPU_intensive_start_routine, NULL) 

  params.sched_priority = 99;
  pthread_attr_setschedparam(&my_rt_att, &params);

  // create one RealTime thread like this:
  pthread_create(&prio_thread, &my_rt_att, the_locking_start_routine, NULL)  //coma was missing

  ...
}

void *the_locking_start_routine(void *arg) {
  ...
  pthread_mutex_lock(&my_mutex);
  // This thread is on the critical section
  // ... (skipped)
  pthread_mutex_unlock(&my_mutex);
  ...
}

... But it doesn't work, I can't have my desired Priority Inversion.

This is what happens:

As I understand, with a scheduller like Linux's CFS, a non-real time thread (SCHED_OTHER) will not run until there isn't any Real Time thread (SCHED_FIFO or SCHED_RR) in runnning state. But I have achieved this threads running simultaneously:

• (L) One non-real time (SCHED_OTHER) thread locking the mutex and consuming CPU
• (M) several Real Time threads (SCHED_FIFO , & priority > 0) CPU intensive and non-waiting to lock the mutex
• (H) One Real Time thread (SCHED_FIFO , & highest priority ) waiting for the lock

There are more Real Time CPU intensive threads (M) running than the amount of CPUs of my system ... but the non-real time thread holding (L) the lock is still consuming CPU and finishes it's work and releases the mutex before the "M" threads finish consuming CPU.

Why isn't the low priority thread preempted, the application dead-locked and I can't get priority inversion?

I'm using g++ 4.5.2 on a Ubuntu Desktop 11.04 with kernel 2.6.38-13.

Answer 1

Re: I'm trying to provoke Priority Inversion on a small C++ program for demonstration purposes but I can't: The low priority thread that holds the mutex is not preempted and keeps running ...

That is the beginning of a priority inversion scenario. A low-priority thread grabs an exclusive resource (e.g. mutex) on which high priority threads then block.

To properly show the consequences of priority inversion, you require, for example, three threads: a low (L), middle (M) and high (H) priority thread.

L locks a mutex, for which H contends. So L is running, H is not. This is bad already: important thread H is waiting for less important thread L to do something.

Now M becomes runnable and is compute-intensive. M doesn't care about the mutex; it is not related to H or L. But M has a higher priority than L and kicks L off the CPU.

So now M continues to execute, preventing L from running. That prevents L from reaching the line of code where it releases the mutex, and that prevents H from getting the mutex.

So a middle priority thread M is running instead of the highest priority thread H.

By blocking L, M is able to block H also: inversion.

See if you can code it up exactly like this.