Understanding the child process of a threaded GHC haskell program

Question

I'm trying to understand how the parent and various child OS threads work in a haskell program compiled with GHC -threaded.

Using

module Main where
import Control.Concurrent
main = do
    threadDelay 9999999999

Compiling with -threaded on ghc 8.6.5, and running with +RTS -N3 for instance, I can see

$ pstree -p 6615 
hello(6615)─┬─{ghc_ticker}(6618)
            ├─{hello:w}(6616)
            ├─{hello:w}(6617)
            ├─{hello:w}(6619)
            ├─{hello:w}(6620)
            ├─{hello:w}(6621)
            ├─{hello:w}(6622)
            └─{hello:w}(6623)

It looks like I get N*2 + 1 of these "hello:w" threads as I vary +RTS -N.

What are these "hello:w" threads, and why are there apparently two per HEC + 1?

And what does ghc_ticker do?

I also noticed on a large real service I'm testing with +RTS -N4 I'm getting e.g. 14 of these "my-service:w" threads, and when under load these process IDs seem to churn (half of them stay alive until I kill the service).

Why 14, and why are half of them spawned and die?

I'd also accept an answer that helped guide me to instrumenting my code to figure out these latter two questions.

Answer 1

The ghc_ticker in spawned at startup, it runs this function. It's purpose is described as

The interval timer is used for profiling and for context switching in the threaded build.

The other *:w threads are workers, they are created whenever there is more work to do (aka Task), but there are no more spare workers, see here

On startup ghc creates one worker per capability, then they are created as needed and reused when possible. It's hard to say why you have 14 workers in -N4 case. I can only guess that they are serving IO manager threads: see here. Let's not forget about FFI also - FFI call may block worker. You can try to put a breakpoint in createOSThread to see why workers are created.

You can read more about scheduler here

ADDED: Hmm, I think I can explain the N*2+1 workers: N workers per capability are created at startup; N more - IO manager event loops, one per capability; plus one IO manager timer thread. Though I'm not sure why the first N workers (created at startup) where not reused for IO manager threads.