Systemd exits with status=209/STDOUT

Question

Scenario

I have a systemd file which I want to run conditionally, but only if the environment variable ISCAPTUREPOD is set to true.

I have a container which runs two services on startup, but there is a special scenario where I only want one of the services to run. I'm passing an environment variable via Kubernetes which I want to use to control whether the second service does or does not start. I have a script named iscapturepod.sh which checks the environment variable and it is part of an ExecStartPre statement. I want the script to succeed when the environment variable ISCAPTUREPOD is set to "True" and fail if it doesn't exist or is set to something other than "True".

Problem:

No matter what I do ExecStartPre fails. I have even tried just making the script say exit 0. That's the only thing in the entire script just because I wanted to force a success. Systemd still fails with status 209/STDOUT.

Moloch capture service:

[Unit]
Description=Moloch Capture
After=network.target

[Service]
Type=simple
Restart=on-failure
StandardOutput=tty
ExecStartPre= /bin/sh -c '/data/moloch/bin/iscapturepod.sh'
ExecStart=/bin/sh -c '/data/moloch/bin/moloch-capture -c MOLOCH_INSTALL_DIR/etc/config.ini ${OPTIONS} >> /data/moloch/logs/capture.log 2>&1'
LimitCORE=infinity
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

The Script

#!/bin/bash

# This script checks to see whether this pod is or is not a capture pod
# Kubernetes will pass the ISCAPTUREPOD variable as an environment variable with
# value True to  those pods meant for capture and False for the viewer pod.
# This allows us to only use one container for both the viewer and capture pods
# The molochcapture service will run this in an ExecStartPre statement. If it
# throws an error this will prevent the molochcapture service from starting

if [[ ! -z "${ISCAPTUREPOD}" ]]; then  
  if [[ "${ISCAPTUREPOD}" == "True" ]]; then    
    echo This is a capture pod
    exit 0
  else    
    echo This is not a capture pod 1>&2    
    exit 1  
  fi
else  
  echo This is not a capture pod 1>$2  
  exit 1
fi

According to this site 0 should be a success. However, even if I change the script to just be exit 0 I still get:

[root@sensor1 /]# systemctl status molochcapture
● molochcapture.service - Moloch Capture
   Loaded: loaded (/usr/lib/systemd/system/molochcapture.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Mon 2019-01-21 12:58:20 UTC; 1s ago
  Process: 281 ExecStartPre=/bin/sh -c /data/moloch/bin/iscapturepod.sh (code=exited, status=209/STDOUT)

Jan 21 12:58:20 sensor1.lan systemd[1]: Failed to start Moloch Capture.
Jan 21 12:58:20 sensor1.lan systemd[1]: Unit molochcapture.service entered failed state.
Jan 21 12:58:20 sensor1.lan systemd[1]: molochcapture.service failed.
Jan 21 12:58:20 sensor1.lan systemd[1]: molochcapture.service holdoff time over, scheduling restart.
Jan 21 12:58:20 sensor1.lan systemd[1]: Stopped Moloch Capture.
Jan 21 12:58:20 sensor1.lan systemd[1]: start request repeated too quickly for molochcapture.service
Jan 21 12:58:20 sensor1.lan systemd[1]: Failed to start Moloch Capture.
Jan 21 12:58:20 sensor1.lan systemd[1]: Unit molochcapture.service entered failed state.
Jan 21 12:58:20 sensor1.lan systemd[1]: molochcapture.service failed.

I have manually checked the script works correctly and has no problems. Kubernetes is passing the environment variable as expected and the script returns "This is a capture pod". I thought maybe it has something to do with systemd not having access to STDOUT, but that's when I tried just exit 0 and it still failed.

Answer 1

I got the service file from a generic Moloch template and modified it. I didn't catch that it has the line StandardOutput=tty. After I noticed that I realized there are multiple types of StandardOutput you can select for service files. They are well documented here: https://www.freedesktop.org/software/systemd/man/systemd.exec.html. Changing the value from tty to inherit resolved the problem for me.

The issue was that I was trying to output to a non existent TTY line which was what was causing the error to be thrown.

My problem was further compounded because systemd does not inherit the environment from the container so the script would accurately show this failure.