Lowest latency notification method between process under Linux

Question

I'm looking for the lowest latency IPC that allow to put one process to sleep and allow other process to wake it.

I'm looking for the lowest latency method. Some possible methods so far:

1. Writing a byte to a pipe and reading it from it.
2. Writing a byte to a socket and reading it from it.
3. Sending a signal (kill) and waiting for it (sigwait)
4. Using sem_post/sem_wait

Any other better ideas?

Any solution that is Linux specific is fine as well.

Answer 1

Generally... There is almost no difference between the OS methods.

Setup:

1. Two processes with affinity to two different CPUs.
2. One process sleeps (nanosleep) for N microseconds measures current time and then notifies other process.
3. Other process wakes measures current time and compares it to the client's time.
4. Average, standard deviation, median and percentile 95 is calculated over 1K samples after warm-up on 100 notifications.
5. OS: Linux 2.6.35 x86_64
6. CPU: Intel i5 M460

Results:

Semaphore (sem_wait/sem_post - kernel - futex):

sleep us     mean             median      %95
       1     4.98 ±18.7         3.78     5.04
      10     4.14 ±14.8         3.54     4.00
     100    20.60 ±29.4        22.96    26.96
    1000    49.42 ±37.6        30.62    78.75
   10000    63.20 ±22.0        68.38    84.38

Signal (kill/sigwait)

sleep us     mean             median      %95
       1     4.69 ±3.8          4.21     5.39
      10     5.91 ±14.8         4.19     7.45
     100    23.90 ±17.7        23.41    35.90
    1000    47.38 ±28.0        35.27    81.16
   10000    60.80 ±19.9        68.50    82.36

Pipe (pipe + write/read)

sleep us     mean             median      %95
       1     3.75 ±5.9          3.46     4.45
      10     4.42 ±3.5          3.84     5.18
     100    23.32 ±25.6        24.17    38.05
    1000    51.17 ±35.3        46.34    74.75
   10000    64.69 ±31.0        67.95    86.80

Socket (socketpair +write/read)

sleep us     mean             median      %95
       1     6.07 ±3.2          5.55     6.78
      10     7.00 ±7.1          5.51     8.50
     100    27.57 ±14.1        28.39    50.86
    1000    56.75 ±25.7        50.82    88.74
   10000    73.89 ±16.8        77.54    88.46

As a reference busy waiting:

sleep us     mean             median      %95
       1     0.17 ±0.5          0.13     0.23
      10     0.15 ±0.3          0.13     0.19
     100     0.17 ±0.3          0.16     0.21
    1000     0.22 ±0.1          0.18     0.35
   10000     0.38 ±0.3          0.30     0.78

Answer 2

Using the same code provided by @Artyom, but in a more morden hardware.

CPU: i9-9900k, closing C/S/P-state and set scaling policy as performance which keeps cores running at the maximum frequency(~5GHz).

OS: Preempt-RT patched Linux with kernel5.0.21, providing better real-time performance.

CPU affinity: two processes apartly running in two isolated cores, which keep away from irrelevant processes and interrupts AMSP.

Results:

Semaphore (sem_wait/sem_post - kernel - futex):

sleep us     mean             minimum  median      %99
       1     1.75 ±0.1          1.60     1.74     1.82
      10     1.76 ±0.0          1.61     1.75     1.83
     100     2.12 ±0.3          1.59     2.24     2.42
    1000     2.46 ±0.3          1.75     2.47     2.56
   10000     2.45 ±0.1          2.11     2.44     2.54

Signal (kill/sigwait)

sleep us     mean            minimum   median      %99
       1     2.15 ±0.2          2.00     2.13     2.22
      10     2.12 ±0.2          1.93     2.11     2.19
     100     2.56 ±0.3          2.00     2.67     2.88
    1000     2.90 ±0.3          2.17     2.90     3.01
   10000     2.94 ±0.5          2.66     2.89     3.03

Pipe (pipe + write/read)

sleep us     mean            minimum   median      %99
       1     2.05 ±0.2          1.88     2.03     2.15
      10     2.06 ±0.3          1.89     2.04     2.17
     100     2.54 ±0.4          1.88     2.63     2.87
    1000     2.98 ±0.3          2.27     2.98     3.09
   10000     2.98 ±0.3          2.69     2.96     3.07

Socket (socketpair +write/read)

sleep us     mean            minimum   median      %99
       1     3.11 ±0.4          2.85     3.09     3.22
      10     3.14 ±0.1          2.92     3.14     3.25
     100     3.66 ±0.5          2.92     3.74     4.01
    1000     4.03 ±0.4          3.28     4.03     4.17
   10000     3.99 ±0.4          3.64     3.96     4.10

As a reference busy waiting:

sleep us     mean            minimum   median      %99
       1     0.07 ±0.1          0.06     0.07     0.07
      10     0.07 ±0.1          0.06     0.07     0.07
     100     0.07 ±0.0          0.06     0.07     0.08
    1000     0.09 ±0.1          0.07     0.08     0.09
   10000     0.09 ±0.1          0.07     0.09     0.09