Linux kernel unresponsive when multiple high priority thread run on multiple cores

Question

Wrote a sample C++ multithreaded program that runs with 10 threads, each thread set to high priority and affinity. Compiled and ran this code on dell machine that has 16 cores, running centos 7 (Linux kernel - 3.10.0-229), disabled hyperthreading. After I ran this code, in few seconds, our Linux machine becomes unresponsive, in the sense that, if I open up Eclipse editor, and save a file or save a file in vi editor those applications hang. Interestingly, once I stop this program / process, then all other applications resume from where they left off. Also I don't see this issue if I remove priority from these 10 threads.

Questions:

1) Out of 16 cores, 6 cores still left on the machine (above cpu usage shows, cpu executed 62.9% user space, and was idle for 37.1%. Interestingly 0% cpu usage in kernel space), so ideally kernel should have used those 6 cores to handle other application, what could be the reason that other application does not able to run? How to resolve this issue without introducing sleep / changing priority?

2) Like to know better approach other than introducing a sleep / waiting on a event (which introduce minimal latency due to kernel context switch) in the thread to achieve parallelism?

Ran top command (top -H):

%Cpu(s):  62.9 us,  0.0 sy,  0.0 ni, 37.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

1107 arun      rt   0   96748   1112    932 R 99.9  0.0   0:25.78 PthreadTest
1115 arun      rt   0   96748   1112    932 R 99.9  0.0   0:24.79 PthreadTest
1118 arun      rt   0   96748   1112    932 R 99.9  0.0   0:22.79 PthreadTest
1120 arun      rt   0   96748   1112    932 R 99.9  0.0   0:20.79 PthreadTest
1123 arun      rt   0   96748   1112    932 R 99.9  0.0   0:18.79 PthreadTest
1117 arun      rt   0   96748   1112    932 R 94.1  0.0   0:23.78 PthreadTest
1119 arun      rt   0   96748   1112    932 R 94.1  0.0   0:21.78 PthreadTest
1122 arun      rt   0   96748   1112    932 R 94.1  0.0   0:19.78 PthreadTest
1124 arun      rt   0   96748   1112    932 R 94.1  0.0   0:17.78 PthreadTest
1125 arun      rt   0   96748   1112    932 R 94.1  0.0   0:16.76 PthreadTest

Code below:

#include <unistd.h>
#include <iostream>
#include <cstdlib>
#include <pthread.h>

using namespace std;

#define NUM_THREADS 10

void *PrintHello(void *threadid)
{
   long tid;
   tid = (long)threadid;

   cout << "Hello World! Thread ID, " << tid << endl;
   while(true)
   {
        continue;
   }
   pthread_exit(NULL);
}

int main ()
{
   pthread_t threads[NUM_THREADS];
   pthread_attr_t  threads_attr[NUM_THREADS];
   struct sched_param     params;
   params.sched_priority = sched_get_priority_max(SCHED_FIFO);
   int rc;
   int i;
   int cpu_num = 0;

   for( i=0; i < NUM_THREADS; i++ ){

      cpu_set_t cpu;
      CPU_ZERO(&cpu);
      CPU_SET(cpu_num, &cpu);
      cout << "main() : creating thread, " << i << "cpu_num : "<<cpu_num<<endl;
      pthread_attr_init(&threads_attr[i]);
      pthread_attr_setscope(&threads_attr[i], PTHREAD_SCOPE_SYSTEM);
      rc = pthread_create(&threads[i], NULL,
                          PrintHello, (void *)i);
      if (rc){
         cout << "Error:unable to create thread," << rc << endl;
         exit(-1);
      }

      sleep(1);


      int ret = pthread_setaffinity_np(threads[i], sizeof(cpu_set_t), &cpu);
      if(ret == 0 && CPU_ISSET(cpu_num, &cpu))
      {
        cout << "Thread " << i << " affinity set " <<endl;
      }


      ret = pthread_setschedparam(threads[i], SCHED_FIFO, &params);
      if(ret == 0)
      {
        cout << "Thread " << i << " priority set " <<endl;
      }
      cpu_num++;
   }


// free attribute and wait for the other threads
   void *status;
   for( i=0; i < NUM_THREADS; i++ )
   {
        rc = pthread_join(threads[i], &status);
        if (rc){
            cout << "Error:unable to join," << rc << endl;
            exit(-1);
        }
        cout << "Main: completed thread id :" << i ;
        cout << "  exiting with status :" << status << endl;
   }

   pthread_exit(NULL);
}

Compile:

g++ -std=c++14 -g -o PthreadTest busywait.cpp -lpthread

Answer 1

The effects of suddenly depriving the kernel of any use of a live core for an unlimited amount of time are unspecified and unknown. Anything attached to that core before exclusive ownership of it was taken, which could include threads that are waiting to be scheduled on it, is forever lost to the system.

DON'T DO THIS!