Priority of kernel modules and SCHED_RR threads

Question

I have an embedded Linux platform (the Beagleboard, running Angstrom Linux) with two devices connected:

• a Laser range finder (Hokuyo UTM 30) connected via USB
• a custom external board connected via SPI

We have a written a Linux kernel module which is responsible for the SPI data transfer. It has an IRQ handler in which spi_async is called which in turn causes an async callback method to be called.

My C++ application consists of three threads:

• a main thread for data processing
• a laser polling thread
• an SPI polling thread

I am experiencing problems which seem to be caused by how the modules described above interact.

• When I switch off the USB device (laser range finder) I receive all SPI messages correctly (1 message every 3ms, message length divided by data rate is <1ms), independent from thread scheduling
• When I switch on the USB device and I run my program with normal thread scheduling (SCHED_OTHER, priority 0, no nice level set) about 1% of the messages is "lost" because the callback method of spi_async is running when the next IRQ occurs (I could handle this case differently in order not to loose the messages, so this is not a big issue.)

• With the USB device turned on and I run the program with SCHED_RR and

  • priority = 10 for main thread
  • priority = 10 for SPI reading thread
  • priority = 4 for USB/Laser polling thread

  then I am loosing 40% of the messages because the IRQ is triggered again before the spi-callback method is called! (I could still maybe find a workaround, but the problem is that I need fast response times which can no longer be reached in this case). I need to use the thread scheduling and the laser device so I am looking for a way to solve this case.

Question 1:

My assumption was that IRQ handlers and the callbacks triggered by spi_async in kernel space have higher priority than any thread running in user space (no matter if SCHED_RR or SCHED_OTHER). This would mean that turning to SCHED_RR in my application shouldn't slow down SPI transfer, but this seems very wrong. Is it?

Question 2:

How can I determine what happens here? Which debugging aids exist? (Or maybe you don't need any further information?) The main question for me is: why do I experience the problems only when the laser device is turned on. Could the USB driver consume so much time?

----- EDIT:

I have made the following observation:

The spi_async's callback calls wake_up_interruptible(&mydata->readq); (with wait_queue_head_t readq;). From the user space (my app) I call a function which results in poll_wait(file, &mydata->readq, wait); When the poll returns the user space calls read().

• When my application runs with SCHED_OTHER I can see that the callback method first finishes before the read() method in my kernel module is entered.
• When my application runs with SCHED_RR read is entered before exiting the callback.

This seems to proof that the priority of the user space threads is higher than the callback method's context's priority. Is there any way to change this behaviour and still have SCHED_RR for my application's threads?

Answer 1

Not all kernel thread have an RT priority. Imagine a periodically waking up thread that needs to do some background work is waking up. You don't want this thread to preemt your RT thread. So I guess your first assumption is wrong.

Based on your other questions :

• your main processing loop receives SPI data through a queue
• the spi processing thread feeds the main processing queue

It seems your main processing thread get in the way of the spi driver thread responsible for the spi data transfer.

Here is what happens :

• an IRQ is fired
• spi_async is called, which means a data transfer is queued, that will be picked up by a thread created by the spi master driver.
• spi master thread compete with your main processing thread, the laser thread, but this kernel thread has not RT priority, so it looses every time one of the RR thread is running.

What you can do is going back to normal scheduling, while playing with the various CONFIG_PREEMPT_ options. Or mess with the spi master driver, to ensure that any delayed work is queued with enough priority. Or even not queued at all.