How does SMP multithreading share memory and interrupts?

Question

I'm doing some work on the input buffers for my kernel, and I had some questions. On Dual Core machines, I know that more than one "process" can be running simultaneously. What I don't know is how the OS and the individual programs work to protect collisions in data.

There are two things I'd like to know on this topic:

(1) Where do interrupts occur? Are they guaranteed to occur on one core and not the other, and could this be used to make sure that real-time operations on one core were not interrupted by, say, file IO which could be handled on the other core? (I'd logically assume that the interrupts would happen on the 1st core, but is that always true, and how would you tell? Or perhaps does each core have its own settings for interrupts? Wouldn't that lead to a scenario where each core could react simultaneously to the same interrupt, possibly in different ways?)

(2) How does the dual core processor handle opcode memory collision? If one core is reading an address in memory at exactly the same time that another core is writing to that same address in memory, what happens? Is an exception thrown, or is a value read? (I'd assume the write would work either way.) If a value is read, is it guaranteed to be either the old or new value at the time of the collision?

I understand that programs should ideally be written to avoid these kinds of complications, but the OS certainly can't expect that, and will need to be able to handle such events without choking on itself.

Answer 1

In x86 processors this is handled by the APIC. You can see details in Intel® 64 and IA-32 Architectures Software Developer's Manual, specifically in volume 3, chapter 9, and in the x2APIC specification.

I'll just give a quick summary in case you don't want to go into all the details.

Interrupts can come from three different sources:

• External pins (in Intel processors up to Core i7 you have LINT0, LINT1, SMI, INIT. I don't know what they're called in Core i7 or AMD or Via processors).
• Bus transactions. This is the main way one thread sends an interrupt to another thread in a modern system. They are referred to as IPIs - Inter-Processor Interrupts.
• Internal events, such as thermal interrupts, event monitors or internal errors.

Each logical processor (thread in an SMT system, core in a non-SMT multi-core system, processor in a non-SMT non-multi-core system) has an APIC. The APIC controls how the logical processor responds to any such interrupts.

In short:

The SMI and INIT pins are always routed to SMI or INIT, respectively.

If the APIC is disabled, LINT0 is routed to INTR, LINT1 is routed to NMI, and IPIs are ignored.

If it's enabled:

• LINT0, LINT1, Thermal events, event monitors and errors each have an entry in the LVT (Logical Vector Table) that specifies if it's masked or not, and if not, what type of interrupt it will be.
• IPIs are handled. IPIs include the type of interrupt (i.e. INTR, SMI, NMI, INIT, SIPI) and the destination. Each logical processor has an APIC-ID, which is . If the IPI's destination matches its ID, it handles the interrupt. Otherwise it ignores it.

If you want details about enabling the APIC, programming the LVT, setting the APIC-ID and sending IPIs, you'll have to look at the manuals I linked to.

Answer 2

The OS gets to set up where interrupts are handled. Linux does load balancing of interrupts, so that they can be handled by both CPUs. Each interrupt handler needs to acquire a lock, to avoid concurrent executions of the same handler on a different CPU, but also to protect from other kernel code running in non-interrupt context and accessing the same data structures. However, I think one can bind execution of a given interrupt on a given CPU.

About question (2): the guarantees are basically the same as given by a SMP machine, i.e. no exception is thrown, and the result depends on who gets to execute/commit the value to memory/commit the value to the shared cache first. You can't anyway rely on the read value - in fact, the guarantees given are much less strong than you expect.

Look on Internet (on Google or Wikipedia) about what a data race is, and start by studying how to write multithreaded code correctly in Java. Studying that made me a lot easier to understand concurrency mechanisms of the Linux kernel.

Or just go for the C/C++ almost "official" memory model FAQ, for Documentation/memory-barriers.txt from the Linux kernel source tree, or for Jeremy Manson's post on the issue. Anyway, I forgot to point out that the value you read has not necessarily been actually written by some processor. For 32-bit values, this is guaranteed by the fact that a 32bit write is atomic. For 64bit values, this is not normally true (I'm not sure about 64bit platforms, but because of portability reasons I don't usually rely on it).

In any case, if you find yourself asking that question, you probably should improve the locking used by your code. Working in kernel, you first need to write your own spinlock/semaphore library to fix this.

When you say "your kernel", it is not clear what you mean, but I think it is unlikely you actually mean "a kernel I'm writing". Anyway, I won't let anyone asking question (2) to run multithreaded programs on my machine :-).

I understand that programs should ideally be written to avoid these kinds of complications, but the OS certainly can't expect that, and will need to be able to handle such events without choking on itself.

The answer to that question is something you need to know to write also userspace multithreaded programs. Well, you don't need to know the exact answer to "which value you read", but just because you can't rely on that, it's implementation-defined even if you write assembly code for a specific processor. Simply because you cannot rely on the relative speed of two parallel threads. Ever.