CLI instruction not executed in Linux kernel module

Question

I'm writing a Linux v3.2 kernel module on an Intel Atom processor (x86_64 with 2 cores). I want to disable a specific IRQ number, but I'm having trouble doing so on Linux.

I'm dual-booting MS-DOS where I can easily disable interrupts in Intel syntax x86 assembly by communicating directly to the 8259 PIC chip:

CLI                ; disable all interrupts
MOV    DX, 0x21    ; set 8259 ioport address
IN     AL, DX      ; store current interrupt mask in AL
AND    AL, 0xDF    ; modify mask to disable IRQ 5
OUT    DX, AL      ; send new mask to 8259
STI                ; reenable interrupts

This works quite well and I'm successfully able to disable specific IRQ numbers.

In Linux, I'm aware that I must use disable_irq macro to disable interrupts, but it seems to have no effect.

#include <linux/interrupt.h>
...
disable_irq(5);    // disable IRQ 5

The disable_irq line is at the beginning of my character driver's open function. However, while the rest of the code in my open function execute as usual when I open my device node, IRQ 5 is left enabled -- it seems like disable_irq had no effect at all.

I wasn't sure whether I'm using the disable_irq macro correctly, so I decided to try straight inline assembly to verify that my logic is correct. I decided to start simple and first try to disable all interrupts:

__asm__("cli");

However, not even this single instruction seems to get executed because all the interrupts are still left enabled.

I'm utterly confused now, why doesn't straight assembly disable the interrupts on Linux? What is the correct way to disable interrupts on Linux?

---

UPDATE: I discovered that disable_irq works only if it is executed after a request_irq. Is this a bug, or expected behavior?

I found a thread that seems to vaguely describe the behavior I'm seeing, but it's dated and I'm not sure it's still relevant for my version of Linux.

---

UPDATE2:

Here's the kernel module I tried on Debian running Linux v3.2.0-4:

#include <linux/module.h>
#include <linux/kernel.h> /* Needed for KERN_INFO */
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/irqflags.h> /* Needed for local_irq_disable et al. */

MODULE_LICENSE("GPL");

static unsigned long flags = 0;

static int __init initialization_routine(void)
{
        local_irq_save(flags);
        local_irq_disable();

        /* __asm__("cli"); */
        /* disable_irq(15); */
        return 0;
}

static void __exit cleanup_routine(void) {
        local_irq_restore(flags);

        /* __asm__("sti"); */
        /* enable_irq(15); */
        return;
}   

module_init(initialization_routine);
module_exit(cleanup_routine);

disable_irq/enable_irq works correctly. I'm not too particularly interested in the plain assembly instructions (it's just odd that they do not work). Moreover, I'm concerned about why local_irq_disable has no observable effect on any of the cores -- i.e. IRQ still show up on all cores.

To check for interrupts I run the following command in my terminal:

$ watch -d -n 0.5 cat /proc/interrupt

Since disable_irq and enable_irq run perfectly now, I suspect I'm simply forgetting some sort of initialization code or perhaps local_irq_disable and related functions are deprecated or do not apply to x86 processors?

Answer 1

I happened onto this question. It's been a year, but not really a good answer yet. I'd like to offer a high-level perspective that seems to be missing about Intel architecture and Linux's architecture. You can dig into the excellent Intel® 64 and IA-32 Architectures Software Developer’s Manual, which Intel shares in PDF form on its website. But here's the information to frame the answer, which is basically, don't try to block a specific vector for all cpus - use spin_locks carefully, and local_irq_disable().

Each processor core has an "interrupt flag" that enables or disables the interruption mechanism on that core only. That's what Linux's local_irq_* routines change (they use the CLI/STI instructions or save, modify, and reload the flags register, changing the IF flag).

Device interrupts are routed to a specific processor core, to its Local APIC (advanced programmable interrupt controller), where an IRQ bit is set for a particular interrupt vector. One of several ways to block a specific interrupt vector from being injected is to set the mask register in the Local APIC (LAPIC) for that core. It can only be done while actually running on that specific core, which makes it tricky to do unless you know your code is running on that core. Generally, this mechanism might be useful if used to take care that nested interrupts don't happen. But the LAPIC can handle the nested interrupt more easily: use the priority masking provided in the LAPIC, since you typically want to block the current interrupt and all lower priority interrupts while running in a handler. Look at the LAPIC PPR register functionality for this. Generally you won't need to write any code to benefit from this, because Linux's interrupt handling is designed around the PPR mechanism and just works.

In general, I would recommend designing your device driver code in a way that would completely avoid the idea of trying to "disable" an interrupt globally. There's no easy way to prevent loss of interrupts "in flight" or other disasters. Instead, use the priority mechanism to deal with interrupts in device interrupt handlers, and construct other code with a careful use of spin_lock or spin_lock_irqsave which as a side effect block interrupts on the cpu executing it.

If the interlocking code is short, you should be able to design the device so you spin_lock_irqsave in the device interrupt handler itself, which safely prevents other cores from touching the device during an interrupt, and vice versa.

Now if you really need to have a randomly selected core block the interrupt for other cores, when it is operating outside an interrupt handler, you need to look at how interrupts are actually delivered to a core's LAPIC. I don't know why you would want to do this - most device drivers don't try to do this, but it might be relevant for some device-specific reason.

Interrupts are delivered to LAPICs over a shared interrupt bus, which allows interrupts to be transmitted to one or more LAPICs in a single transaction. (the multicast modes are rarely, if ever, used for devices, so I won't explain them here - see Intel's manual). The shared interrupt bus either sends interrupts to an IOAPIC device or over the Front-Side-Bus directly to a LAPIC, addressed via its APIC ID, and specifying the vector number to be used. (the FSB mechanism can be used directly by more modern devices - PCI Express and the HPET, but most legacy devices use the IOAPIC.)

To block a specific vector globally across all processors, you can do one of three things, conceptually: a) reprogram the IOAPIC or PCI Express register that the device uses to set a "mask" bit. However, that's a lot of stuff, and may require stopping all the other cpu cores that might be getting an interrupt or other magic. You might use this when starting or stopping the device, once. b) do something with the shared IDT to capture the interrupt temporarily, and then forward it to the right handler later. This is also complicated and dangerous. c) Change the device state to not generate interrupts at all. Most devices have the ability to not generate interrupts.

However, one must remember that there may be one or more interrupts "in flight" while trying to do this global suppression.