Do Linux kernel processes multithread?

Question

For Linux user space processes it seems pretty easy to determine which processes are multithreading. You can use ps -eLf and look at the NLWP value for the number of threads, which also corresponds to the 'Threads:' value in /proc/$pid/status. Apparently back in the day of LinuxThreads the implementation was not POSIX compliant. But This stackoverflow answer says "POSIX.1 requires threads share a same process ID" which is apparently rectified in NPTL. So with NPTL it allows nifty displays of threads with commands like ps -eLf because the threads all share the same PID, and you can verify that under /proc/$pid/task/ and see all the thread subfolders belonging to that "parent" process.

I can't find similar thread groupings under a "parent" process for kernel processes spawned by kthreadd, and I suspect an implementation difference since a comment under this answer says "You can not use POSIX Threads in kernel-space" and the nifty thread grouping is a POSIX feature. Thus with ps -eLf I never see multiple threads listed for kernel processes created by kthreadd which have the square brackets around it, like [ksoftirqd/0] or [nfsd], unlike user-space processes created by init.

From the man page for pthreads (which is used in the user space):

       A single process can contain multiple threads, all of which are
       executing the same program.  These threads share the same global
       memory (data and heap segments), but each thread has its own stack
       (automatic variables).

This however is precisely what I do not see for kernel "threads", in terms of one process containing multiple threads.

In short, I never see any of the processes listed by 'ps' that are children of kthreadd having a NLWP (Threads) value higher than one, which makes me wonder if any kernel process forks/parallelizes and multithreads like user-space programs do (with pthreads). Where do the implementations differ?

Practical example: Output from ps auxf for the NFS processes.

root         2  0.0  0.0      0     0 ?        S    Jan12   0:00 [kthreadd]
root      1546  0.0  0.0      0     0 ?        S    Jan12   0:00  \_ [lockd]
root      1547  0.0  0.0      0     0 ?        S    Jan12   0:00  \_ [nfsd4]
root      1548  0.0  0.0      0     0 ?        S    Jan12   0:00  \_ [nfsd4_callbacks]
root      1549  0.0  0.0      0     0 ?        S    Jan12   2:40  \_ [nfsd]
root      1550  0.0  0.0      0     0 ?        S    Jan12   2:39  \_ [nfsd]
root      1551  0.0  0.0      0     0 ?        S    Jan12   2:40  \_ [nfsd]
root      1552  0.0  0.0      0     0 ?        S    Jan12   2:47  \_ [nfsd]
root      1553  0.0  0.0      0     0 ?        S    Jan12   2:34  \_ [nfsd]
root      1554  0.0  0.0      0     0 ?        S    Jan12   2:39  \_ [nfsd]
root      1555  0.0  0.0      0     0 ?        S    Jan12   2:57  \_ [nfsd]
root      1556  0.0  0.0      0     0 ?        S    Jan12   2:41  \_ [nfsd]

By default when you start the rpc.nfsd service (at least with the init.d service script) it spawns 8 processes (or at least they have PIDs). If I wanted to write a multi-threaded version of NFS, which is implemented as a kernel module, with those nfsd "processes" as a frontend, why couldn't I group the default 8 different nfsd processes under one PID and have 8 threads running under it, versus (as is shown - and as is different than user space processes) eight different PIDs?

NSLCD is an example of a user space program that uses multithreading by contrast:

UID        PID  PPID   LWP  C NLWP STIME TTY          TIME CMD
nslcd     1424     1  1424  0    6 Jan12 ?        00:00:00 /usr/sbin/nslcd
nslcd     1424     1  1425  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd
nslcd     1424     1  1426  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd
nslcd     1424     1  1427  0    6 Jan12 ?        00:00:27 /usr/sbin/nslcd
nslcd     1424     1  1428  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd
nslcd     1424     1  1429  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd

The PID is the same but the LWP is unique per thread.

Update on the function of kthreadd

From this stackoverflow answer:

kthreadd is a daemon thread that runs in kernel space. The reason is that kernel needs to some times create threads but creating thread in kernel is very tricky. Hence kthreadd is a thread that kernel uses to spawn newer threads if required from there . This thread can access userspace address space also but should not do so . Its managed by kernel...

And this one:

kthreadd() is main function (and main loop) of daemon kthreadd which is a kernel thread daemon, the parent of all other kernel threads.

So in the code quoted, there is a creation of request to kthreadd daemon. To fulfill this request kthreadd will read it and start a kernel thread.

Answer 1

There is no concept of a process in the kernel, so your question doesn't really make sense. The Linux kernel can and does create threads that run completely in kernel context, but all of these threads run in the same address space. There's no grouping of similar threads by PID, although related threads usually have related names.

If multiple kernel threads are working on the same task or otherwise sharing data, then they need to coordinate access to that data via locking or other concurrent algorithms. Of course the pthreads API isn't available in the kernel, but one can use kernel mutexes, wait queues, etc to get the same capabilities as pthread mutexes, condition variables, etc.

Calling these contexts of execution "kernel threads" is a reasonably good name, since they are closely analogous to multiple threads in a userspace process. They all share the (kernel's) address space, but have their own execution context (stack, program counter, etc) and are each scheduled independently and run in parallel. On the other hand, the kernel is what actually implements all the nice POSIX API abstractions (with help from the C library in userspace), so internal to that implementation we don't have the full abstraction.