How to know the number of active threads in Puma

Question

I am trying to see the number of active puma threads on my server.

I can not see it through ps:

$ ps aux | grep puma
healthd   2623  0.0  1.8 683168 37700 ?        Ssl  May02   5:38 puma 2.11.1 (tcp://127.0.0.1:22221) [healthd]  
root      8029  0.0  0.1 110460  2184 pts/0    S+   06:34   0:00 grep --color=auto puma
root     18084  0.0  0.1  56836  2664 ?        Ss   May05   0:00 su -s /bin/bash -c puma -C /opt/elasticbeanstalk/support/conf/pumaconf.rb webapp
webapp   18113  0.0  0.8  83280 17324 ?        Ssl  May05   0:04 puma 2.16.0 (unix:///var/run/puma/my_app.sock) [/]                                                               
webapp   18116  3.5  6.2 784992 128924 ?       Sl   May05 182:35 puma: cluster worker 0: 18113 [/] 

As in the configuration I have:

threads 8, 32

I was expecting to see at least 8 puma threads?

Answer 1

To quickly answer the question, the number of threads used by a process running on a given PID, can be obtained using the following :

% ps -h -o nlwp <pid>

This will just return you the total number of threads used by the process. The option -h removes the headers and the option -o nlwp formats the output of ps such that it only outputs the Number of Light Weight Processes (NLWP) or threads. Example, when only a single process puma is running and its PID is obtained with pgrep, you get:

% ps -h -o nlwp $(pgrep puma)
   4

What is the difference between process, thread and light-weight process?

This question has been answered already in various places [See here, here and the excellent geekstuff article]. The quick, short and ugly version is :

• a process is essentially any running instance of a program.

• a thread is a flow of execution of the process. A process containing multiple execution-flows is known as multi-threaded process and shares its resources amongst its threads (memory, open files, io, ...). The Linux kernel has no knowledge of what threads are and only knows processes. In the past, multi-threading was handled on a user level and not kernel level. This made it hard for the kernel to do proper process management.

• Enter lightweight processes (LWP). This is essentially the answer to the issue with threads. Each thread is considered to be an LWP on kernel level. The main difference between a process and an LWP is that the LWP shares resources. In other words, an Light Weight Process is kernel-speak for what users call a thread.

Can ps show information about threads or LWP's?

The ps command or process status command provides information about the currently running processes including their corresponding LWPs or threads. To do this, it makes use of the /proc directory which is a virtual filesystem and regarded as the control and information centre of the kernel. [See here and here].

By default ps will not give you any information about the LWPs, however, adding the option -L and -m to the command generally does the trick.

man ps :: THREAD DISPLAY

   H      Show threads as if they were processes.
   -L     Show threads, possibly with LWP and NLWP columns.
   m      Show threads after processes.
   -m     Show threads after processes.
   -T     Show threads, possibly with SPID column.

For a single process puma with pid given by pgrep puma

% ps -fL $(pgrep puma)
UID        PID  PPID   LWP  C NLWP STIME TTY      STAT   TIME CMD
kvantour  2160  2876  2160  0    4 15:22 pts/39   Sl+    0:00 ./puma
kvantour  2160  2876  2161 99    4 15:22 pts/39   Rl+    0:14 ./puma
kvantour  2160  2876  2162 99    4 15:22 pts/39   Rl+    0:14 ./puma
kvantour  2160  2876  2163 99    4 15:22 pts/39   Rl+    0:14 ./puma

however, adding the -m option clearly gives a nicer overview. This is especially handy when multiple processes are running with the same name.

% ps -fmL $(pgrep puma)
UID        PID  PPID   LWP  C NLWP STIME TTY      STAT   TIME CMD
kvantour  2160  2876     -  0    4 15:22 pts/39   -      0:44 ./puma
kvantour     -     -  2160  0    - 15:22 -        Sl+    0:00 -     
kvantour     -     -  2161 99    - 15:22 -        Rl+    0:14 -     
kvantour     -     -  2162 99    - 15:22 -        Rl+    0:14 -     
kvantour     -     -  2163 99    - 15:22 -        Rl+    0:14 -     

In this example, you see that process puma with PID 2160 runs with 4 threads (NLWP) having the ID's 2160--2163. Under STAT you see two different values Sl+ and 'Rl+'. Here the l is an indicator for multi-threaded. S and R stand for interruptible sleep (waiting for an event to complete) and respectively running. So we see that 3 of the 4 threads are running at 99% CPU and one thread is sleeping. You also see the total accumulated CPU time (44s) while a single thread only runs for 14s.

Another way to obtain information is by directly using the format specifiers with -o or -O.

man ps :: STANDARD FORMAT SPECIFIERS

   lwp    lightweight process (thread) ID of the dispatchable
          entity (alias spid, tid).  See tid for additional
          information.  Show threads as if they were processes.
   nlwp   number of lwps (threads) in the process.  (alias thcount).

So you can use any of lwp,spid or tid and nlwp or thcount.

If you only want to get the number of threads of a process called puma, you can use :

% ps -o nlwp $(pgrep puma)
NLWP
   4

or if you don't like the header

% ps -h -o nlwp $(pgrep puma)
   4

You can get a bit more information with :

% ps -O nlwp $(pgrep puma)
PID   NLWP S TTY          TIME COMMAND
19304    4 T pts/39   00:00:00 ./puma

Finally, you can combine the flags with ps aux to list the threads.

 % ps aux -L
USER       PID   LWP %CPU NLWP %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
kvantour  1618  1618  0.0    4  0.0  33260  1436 pts/39   Sl+  15:17   0:00 ./puma
kvantour  1618  1619 99.8    4  0.0  33260  1436 pts/39   Rl+  15:17   0:14 ./puma
kvantour  1618  1620 99.8    4  0.0  33260  1436 pts/39   Rl+  15:17   0:14 ./puma
kvantour  1618  1621 99.8    4  0.0  33260  1436 pts/39   Rl+  15:17   0:14 ./puma
...

Can top show information about threads or LWP's?

top has the option to show threads by hitting H in the interactive mode or by launching top with top -H. The problem is that it lists the threads as processes (similar to ps -fH).

% top
top - 09:42:10 up 17 days, 3 min,  1 user,  load average: 3.35, 3.33, 2.75
Tasks: 353 total,   3 running, 347 sleeping,   3 stopped,   0 zombie
%Cpu(s): 75.5 us,  0.6 sy,  0.5 ni, 22.6 id,  0.0 wa,  0.0 hi,  0.8 si,  0.0 st
KiB Mem : 16310772 total,  8082152 free,  3662436 used,  4566184 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 11363832 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  868 kvantour  20   0   33268   1436   1308 S 299.7  0.0  46:16.22 puma
 1163 root      20   0  920488 282524 258436 S   2.0  1.7 124:48.32 Xorg
 ...

Here you see that puma runs at about 300% CPU for an accumulated time of 46:16.22. There is, however, no indicator that this is a threaded process. The only indicator is the CPU usage, however, this could be below 100% if 3 threads are "sleeping"? Furthermore, the status flag states S which indicates that the first thread is asleep. Hitting H give you then

% top -H
top - 09:48:30 up 17 days, 10 min,  1 user,  load average: 3.18, 3.44, 3.02
Threads: 918 total,   5 running, 910 sleeping,   3 stopped,   0 zombie
%Cpu(s): 75.6 us,  0.2 sy,  0.1 ni, 23.9 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 16310772 total,  8062296 free,  3696164 used,  4552312 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 11345440 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  870 kvantour  20   0   33268   1436   1308 R 99.9  0.0  21:45.35 puma
  869 kvantour  20   0   33268   1436   1308 R 99.7  0.0  21:45.43 puma
  872 kvantour  20   0   33268   1436   1308 R 99.7  0.0  21:45.31 puma
 1163 root      20   0  920552 282288 258200 R  2.0  1.7 124:52.05 Xorg 
  ...

Now we see only 3 threads. As one of the Threads is "sleeping", it is way down the bottom as top sorts by CPU usage.

In order to see all threads, it is best to ask top to display a specific pid (for a single process):

% top -H -p $(pgrep puma)
top - 09:52:48 up 17 days, 14 min,  1 user,  load average: 3.31, 3.38, 3.10
Threads:   4 total,   3 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 75.5 us,  0.1 sy,  0.2 ni, 23.6 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem : 16310772 total,  8041048 free,  3706460 used,  4563264 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 11325008 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  869 kvantour  20   0   33268   1436   1308 R 99.9  0.0  26:03.37 puma
  870 kvantour  20   0   33268   1436   1308 R 99.9  0.0  26:03.30 puma
  872 kvantour  20   0   33268   1436   1308 R 99.9  0.0  26:03.22 puma
  868 kvantour  20   0   33268   1436   1308 S  0.0  0.0   0:00.00 puma

When you have multiple processes running, you might be interested in hitting f and toggle PGRP on. This shows the Group PID of the process. (PID in ps where PID in top is LWP in ps).

How do I get the thread count without using ps or top?

The file /proc/$PID/status contains a line stating how many threads the process with PID $PID is using.

% grep Threads /proc/19304/status
Threads:        4

General comments

• It is possible that you do not find the process of another user and therefore cannot get the number of threads that process is using. This could be due to the mount options of /proc/ (hidepid=2).

Used example program:

#include <omp.h>
#include <stdio.h>
#include <stdlib.h>

int main (int argc, char *argv[]) {
char c = 0;
#pragma omp parallel shared(c)   {
    int i = 0;
    if (omp_get_thread_num() == 0) {
      printf("Read character from input : ");
      c = getchar();
    } else {
      while (c == 0) i++;
      printf("Total sum is on thread %d : %d\n", omp_get_thread_num(), i);
    }
  }
}

compiled with gcc -o puma --openmp

Answer 2

If you are just looking for number of threads that are spawned by the process, you can see the number of task folders created under /proc/[pid-of-process]/task because each thread creates a folder under this path. So counting the number of folders would be sufficient.

In fact the ps utility itself reads the information from this path, a file /proc/[PID]/cmdline which is represented in a more readable way.

From Linux Filesystem Hierarchy

/proc is very special in that it is also a virtual file-system. It's sometimes referred to as a process information pseudo-file system. It doesn't contain 'real' files but run-time system information (e.g. system memory, devices mounted, hardware configuration, etc). For this reason it can be regarded as a control and information center for the kernel. In fact, quite a lot of system utilities are simply calls to files in this directory.

All you need to get the PID of the process puma, use ps or any utility of your choice

ps aux | awk '/[p]uma/{print $1}'

or more directly use pidof(8) - Linux man page which gets you the PID directly given the process name as input

pidof -s puma

Now that you have the PID to count the number of task/ folders your process had created use the find command

find /proc/<PID>/task -maxdepth 1 -type d -print | wc -l