SCHED_FIFO thread is preempted by SCHED_OTHER thread in Linux

Question

I have written a test program to test out SCHED_FIFO. I have learnt that SCHED_FIFO cannot be preempted by SCHED_OTHER threads. But I couldn't explain the results obtained when same program is run multiple times.

/* Includes */
#include <unistd.h>     /* Symbolic Constants */
#include <sys/types.h>  /* Primitive System Data Types */ 
#include <errno.h>      /* Errors */
#include <stdio.h>      /* Input/Output */
#include <stdlib.h>     /* General Utilities */
#include <pthread.h>    /* POSIX Threads */
#include <string.h>     /* String handling */
#include <sched.h>

/* prototype for thread routine */
void print_message_function ( void *ptr );
void print_message_function1 ( void *ptr );

/* struct to hold data to be passed to a thread
 * this shows how multiple data items can be passed to a thread
 */
typedef struct str_thdata
{
    int thread_no;
    int thread_value;
    char message[100];
} thdata;

int main()
{
    pthread_t thread1, thread2;  /* thread variables */
    thdata data1, data2;         /* structs to be passed to threads */

    /* initialize data to pass to thread 1 */
    data1.thread_no = 1;
    data1.thread_value = 0;
    strcpy(data1.message, "Hello!");

    /* initialize data to pass to thread 2 */
    data2.thread_no = 2;
    data2.thread_value = 10000;
    strcpy(data2.message, "Hi!");

    /* create threads 1 and 2 */    
    pthread_create (&thread1, NULL, (void *) &print_message_function, (void *) &data1);
    pthread_create (&thread2, NULL, (void *) &print_message_function1, (void *) &data2);

    /* Main block now waits for both threads to terminate, before it exits
     * If main block exits, both threads exit, even if the threads 
     * have not finished their work 
     */ 
    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);
          
    /* exit */  
    exit(0);
} /* main() */

/** 
* print_message_function is used as the start routine for the threads used
* it accepts a void pointer 
**/
void print_message_function ( void *ptr )
{
    thdata *data;            
    data = (thdata *) ptr;  /* type cast to a pointer to thdata */

    struct sched_param param;
    //int priority=10;
    /* sched_priority will be the priority of the thread */
    //param.sched_priority = priority;
    /* only supported policy, others will result in ENOTSUP */

    int policy = SCHED_OTHER;
    /* scheduling parameters of target thread */
    pthread_setschedparam(pthread_self(), policy, &param);
    printf("Thread %d says sched policy  %d \n", data->thread_no, SCHED_OTHER);
    pthread_getschedparam(pthread_self(),&policy,&param);

    printf("Thread %d says %s  %d \n", data->thread_no, data->message,policy);

    int i=0;
    /* do the work */
    printf("Thread %d says %s %d \n", data->thread_no, data->message,(int)pthread_self());
    for(i=0;i<100;i++) 
        printf("Thread %d says %d \n", data->thread_no,data->thread_value++);
    pthread_exit(0); /* exit */
} /* print_message_function ( void *ptr ) */

void print_message_function1 ( void *ptr )
{
    thdata *data;            
    data = (thdata *) ptr;  /* type cast to a pointer to thdata */

    struct sched_param param;
    int priority=10;
    /* sched_priority will be the priority of the thread */
    param.sched_priority = priority;
    /* only supported policy, others will result in ENOTSUP */

    int policy = SCHED_FIFO;
    /* scheduling parameters of target thread */
    pthread_setschedparam(pthread_self(), policy, &param);
    printf("Thread %d says sched policy %d \n", data->thread_no, SCHED_FIFO);

    pthread_getschedparam(pthread_self(),&policy,&param);

    printf("Thread %d says %s  %d \n", data->thread_no, data->message,policy);

    int i=0;
    /* do the work */
    printf("Thread %d says %s  %d \n", data->thread_no, data->message,(int)pthread_self());
    for(i=0;i<100;i++)
        printf("Thread %d says %d \n", data->thread_no,data->thread_value++);
    pthread_exit(0); /* exit */
} /* print_message_function ( void *ptr ) */

I have got unexpected results in multiple runs where I have seen SCHED_FIFO is preempted by SCHED_OTHER thread, i.e. as per program, thread 2 is in FIFO mode, while thread 1 is SCHED_OTHER mode. I have seen multiple times where thread2 is preempted by thread1.

Can someone help me out in finding the issue?

Answer 1

You probably have these sysctl settings in effect, which are default values:

kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000

This means that real time threads are allowed to hog only 95% of every 1 second period.

Also see: Can't provoke Priority Inversion in C++

Answer 2

Also whenever your thread blocks on IO, perhaps from the printf statements, another thread may be scheduled.