Windows 11+ thread affinities spanning multiple processor groups (explicitly)

Question

I'm trying to figure out how to properly set thread affinities on Windows 11 (or Server 2022, but I'll just say 11 from now on). A short explanation:

Windows 7 through 10 supported more than 64 CPU cores by introducing Processor Groups. In those previous versions one could only set the affinity of a thread to CPUs within a single processor group. It was possible to select groups other than the first one, but it was not possible to span more than one group.

In Windows 11 Microsoft changed things and introduced a new default that the affinity for each thread would span the entire available set of CPU cores. Fantastic. However, once I want to manually change the affinity of a thread, I don't see a possibility of spanning multiple processor groups explicitly (instead of relying on the implicit default that spans all).

A couple of example use cases why that may be relevant:

• Excluding a single core (or a low number of cores) from the affinity of some threads, but keeping all others (because more important things are set to run on those small number of cores).
• If a system has a lot of groups (e.g. 256 cores and at least 4 groups), to just span a subset of the processor groups, instead of all of them.

I can use SetThreadGroupAffinity the same as in Windows 7 through 10 to explicitly set the thread affinity to a subset given by a single processor group. To quote the documentation (emphasis added):

Starting with Windows 11 and Windows Server 2022, on a system with more than 64 processors, process and thread affinities span all processors in the system, across all processor groups, by default. The SetThreadGroupAffinity function restricts a thread's affinity to the processors over the single processor group specified by the given GroupAffinity. This group will also become the thread's primary group.

If I use the even older function SetThreadAffinityMask on Windows 11 it will similarly restrict the affinity of the thread to the selected threads, but just within the primary processor group. To quote the documentation again:

Starting with Windows 11 and Windows Server 2022, on a system with more than 64 processors, process and thread affinities span all processors in the system, across all processor groups, by default. The dwThreadAffinityMask must specify processors in the thread's current primary group.

But there doesn't appear to be any new function that has been added to Windows 11 that can explicitly set the CPU affinity mask of a thread to span multiple processor groups - the only way to have that is to rely on the default, but I can't find any way of doing that explicitly. (I haven't even found a way to revert to the defaults entirely.)

What is new in Windows 11 are additional functions in the CPU Sets subsystem (which was added in Windows 10 initially), but those aren't hard affinities to specific cores - those are just suggestions that the scheduler may ignore. So that doesn't help me when I want to modify the explicit (hard) affinities of a process.

Am I missing something? Or is this something that Microsoft has completely overlooked?

Answer 1

I've now played around with SetThreadSelectedCpuSetMasks() a bit and found that it does in fact solve my problem, but with some caveats.

Windows starting with version 11 appears to have two completely different affinity APIs: the traditional ones (that can only apply to at most one processor group), or the new ones defined by SetThreadSelectedCpuSetMasks(). I've observed the following behavior:

• If both are set Windows takes effectively an AND mask of the affinities. So if for example the SetThreadGroupAffinity() mask is set to 0x03 (CPUs 0 and 1) for group 0 and the SetThreadSelectedCpuSetMasks() mask is set to to 0x06 (CPUs 1 and 2) for group 0, the result is that the thread will only run on CPU 1.
  • However, if there is no overlap between both masks, then only the SetThreadGroupAffinity() (or equivalently SetThreadAffinityMask()) mask is applied, and the mask set by SetThreadSelectedCpuSetMasks() is ignored. (It can still be re-read via GetThreadSelectedCpuSetMasks() and there is no indication purely from that API that there would be an issue there.)
• In contrast to what's written in the documentation, the SetThreadSelectedCpuSetMasks() still appears to be a hard mask (other than the "will be ignored fully" logic when there's no overlap with the "traditional" mask), i.e. not just a suggestion to the scheduler. I've never observed a thread being run on a core that it wasn't supposed to with the new-style affinities set this way.
• Setting either mask will have no effect on the other mask. For example, calling SetThreadSelectedCpuSetMasks() will have no effect on GetThreadGroupAffinity(), and similarly vice-versa.
• I still haven't found anything in the API to undo the effect of SetThreadGroupAffinity()/SetThreadAffinityMask() restricting the affinity mask to just a single processor group.

To summarize, I think the following logic is probably the best way of doing things:

• On Windows before 7 use SetThreadAffinityMask() (it doesn't support more than 64 CPUs anyway) (Probably not relevant anymore nowadays, just wanted to mention it for completeness sake.)
• On Windows 7 through 10 use SetThreadGroupAffinity() to select a single processor group and assign the affinity there. (The OS simply can't schedule threads across processor groups.)
• On Windows 11 onwards never use the SetThreadAffinityMask()/SetThreadGroupAffinity() APIs to set the thread affinity, because we have to rely on the default behavior (without such a call) that all threads can be scheduled across all processor groups. Instead only use the SetThreadSelectedCpuSetMasks() API to set affinities. And pray that no third-party library accidentally calls SetThreadAffinityMask()/SetThreadGroupAffinity() (or even worse SetProcessAffinityMask() for that matter), because that'll undo the default behavior of Windows 11 to schedule across all groups, and there appears to be no way back.

Side note: for programs that still need to run on Windows 10, I intend to use LoadLibraryA("kernel32.dll"); and GetProcAddress(library, "SetThreadSelectedCpuSetMasks"); to a) determine if those calls are supported (GetProcAddress will return nullptr on Windows 10, but not on 11, so it's a good way of detecting this feature) and b) to invoke them from a program compiled with a lower _WIN32_WINNT than required for Windows 11.

I hope this helps other people who might run into the same issue.

I'll leave this question open in case anyone still has an idea of an API function that can be called to get rid of the "restrict to a single processor group" behavior of SetThreadGroupAffinity()/SetThreadAffinityMask() and return to the default behavior regarding that mask. (In that case I would use that call before calling SetThreadSelectedCpuSetMasks() to ensure that only the latter call is relevant to the thread affinities, regardless of what was previously called in the process.)