get monit to alert first and restart later

Question

I would like to handle a kind of chain action in monit.

• check for a process and alert immediately.
• restart process after a num of cycles.

My tries (so far):

check process myprocess with pidfile /run/my.pid
  start program = "/path/to/binary start" with timeout 60 seconds
  stop program = "/path/to/binary stop" with timeout 60 seconds
  if not exist for 3 cycles then restart
  if not exist then alert
  if 3 restarts within 3 cycles then timeout

Does not alert and keeps in state "running" on failing PID but restarts after the 3 cycles.

check process myprocess with pidfile /run/my.pid
  start program = "/path/to/binary start" with timeout 60 seconds
  stop program = "/path/to/binary stop" with timeout 60 seconds
  if not exist for 3 cycles then restart
  if children < 1 for 1 cycles then alert
  if 3 restarts within 3 cycles then timeout

No alert of children < 1 but restart afer 5.

monit.log

[CEST Aug  1 15:09:30] error    : 'myprocess' process is not running

monit summary

Process 'myprocess'            Running

Here ist monit -v part:

Existence      = if does not exist 3 times within 3 cycle(s) then restart else 
                 if succeeded 1 times within 1 cycle(s) then alert
Pid            = if changed 1 times within 1 cycle(s) then alert
Ppid           = if changed 1 times within 1 cycle(s) then alert
Children       = if less than 1 1 times within 1 cycle(s) then alert else if 
                 succeeded 1 times within 1 cycle(s) then alert
Timeout        = If restarted 3 times within 3 cycle(s) then unmonitor

So the question: is it possible to send an alert and change the status to 'not running' within 1 cycle and restart after 3?

Answer 1

EDIT (IMPORTANT): See comments below for newer (as per Feb. 2019) versions of Monit, where this behaviour has been improved.

---

This line:

if does not exist for 3 cycles then restart

Means the following:

Do not perform any action until you have checked 3 times that the service does not exist, then restart it. This behaviour is described in monit's documentation as Failure Tolerance:

FAILURE TOLERANCE

By default the action is executed if it matches and the service set in an error state. However, you can require a test to fail more than once before the error event is triggered and the service state changed to failed. This is useful to avoid getting alerts on spurious errors, which can happen, especially with network tests.

Syntax:

FOR CYCLES ... or:

[TIMES WITHIN] CYCLES ...

Accordingly, Monit wont change the service's status until it fails within the next X cycles. In order to confirm this statement, just remove the fault tolerance for this service and use only:

if does not exist then alert

stop manually the service and confirm that the command

monit status

shows now the status "Does not exist" as soon as you stop it.

So, back to your questions:

1. Yes, it is possible to send an alert (per Email) within 1 cycle. For that, you need to define the option "if does not exist then alert" for that service and setup Email-Alerts correctly. Assuming you would like to use an external eMail-Server, you need to define at least two lines (Configuration Example with gmail):

SMTP SERVER CONFIGURATION

set mailserver smtp.gmail.com PORT 587 USERNAME "[email protected]" PASSWORD "xxxxx" using TLSV1 with timeout 30 seconds

(Be aware that in gmail you must activate the access for "unsecure" apps in order to allow monit to use the stmp service)

and

EMAIL RECIPIENT

set alert [email protected]

both in the file /etc/monit/monitrc. Refer to the official documentation for more information about these two lines.

2. As far as the documentation tells, it is not possible to update the status of the service inmediately if a fault tolerance (perform action after X cycles) is defined. But you can still define alerts to be sent inmediately and restart the service within the desired cycles.

References:

Monit's documentation: https://mmonit.com/monit/documentation/monit.html

Hope it helps!

Regards