tickless kernel , isolcpus,nohz_full,and rcu_nocbs

Question

I have add "isolcpus=3 nohz_full=3 rcu_nocbs=3" in grub.conf in RedHat 7.1 , kernel: linux 3.10.0-229 kernel and according to http://www.breakage.org/2013/11/15/nohz_fullgodmode/ I also execute the following command :

cat /sys/bus/workqueue/devices/writeback/cpumask
f
echo 1 > /sys/bus/workqueue/devices/writeback/cpumask

cat /sys/bus/workqueue/devices/writeback/numa
1
echo 0 > /sys/bus/workqueue/devices/writeback/numa

The box has only 4 cpu cores , I run the following shell :

watch -d 'cat /proc/interrupts'

look like work perfect , only cpu0 Local timer interrupts has 2000 per 2 secs, the else cpu 1 to cpu 3 has less than 10 per 2 secs .

and then I test the following source :

void *Thread2(void *param)
{
    pthread_detach(pthread_self());
    while( 1 ){
        sleep( 100000 ) ;
    }
}

void *Thread1(void *param)
{
    pthread_detach(pthread_self());
    while( 1 ){
        ;
    }
}

int main(int argc, char** argv)
{
    pthread_t tid ;
    pthread_create(&tid , NULL, Thread1, (void*)(long)3);
    pthread_create(&tid , NULL, Thread2, (void*)(long)3);

    while( 1 )
        sleep( 5 ) ;
}

and run it by :

taskset -c 3 ./x1.exe

watch the output in :

watch -d 'cat /proc/interrupts'

this time , cpu 3 get 10~30 Local timer interrupts per 2 secs , look fine, then I try to run 2 thread1 by :

pthread_create(&tid , NULL, Thread1, (void*)(long)3);
pthread_create(&tid , NULL, Thread1, (void*)(long)3);

then again run it :

taskset -c 3 ./x1.exe

then I watch the core 3 has the same Local timer interrupts with core 0 , it is 2000 interrupts per 2 secs .

May I ask , why 2 very busy thread1 will cause core 3 has much more timer interrupts ?! what cause this happened ?! and how to modify it if it can be ?!

Answer 1

In the second case, Kernel needs to schedule 2 cpu bound tasks on core 3 and the dynamic ticks configuration is applicable only when there is exactly one runnable task. I thought SCHED_FIFO would stop these interrupts (and so I started answering), but that isn't yet implemented as per https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt

There is no way to change this behaviour except scheduling threads on different CPUs. You can always hack the kernel to achieve what you need.