How to run a specific program before systemd's watchdog stops a service

Question

I have a program which is run by systemd with a service file like this:

[Unit]
Description=...

[Service]
Type=notify
ExecStart=/usr/sbin/myprogram
WatchdogSec=1
KillMode=process
KillSignal=SIGTERM
Restart=always

It sends the respective signal to the watchdog regularly. From time to time, the program seems to hang and is terminated by the watchdog, then restarts. Before the watchdog terminates it, I'd like to capture some information from the program by executing a command or running some other script (e.g. run gdb -p <PID> --batch -ex 'thread apply all backtrace'). How would I do this?

Answer 1

Add a ExecStop= to your service.

[Service]
ExecStart=....
ExecStop=/path/to/SomeOtherProgram
....

According to systemd manual, if ExecStop option is available, it will run that first, then if the process under ExecStart is still available after this, it will run the KillMode.

ExecStop= Commands to execute to stop the service started via ExecStart=. This argument takes multiple command lines, following the same scheme as described for ExecStart= above. Use of this setting is optional. After the commands configured in this option are run, it is implied that the service is stopped, and any processes remaining for it are terminated according to the KillMode= setting (see systemd.kill(5)). If this option is not specified, the process is terminated by sending the signal specified in KillSignal= when service stop is requested. Specifier and environment variable substitution is supported (including $MAINPID, see above).

EDIT

As in the comment below, this solution may not work for Watchdog option in the service file.