Airflow systemd scheduler not working

Question

I got the webserver to work properly, and my airflow-scheduler.service file starts the scheduler and it finds my dags etc. However, tasks are not running:

I see an error message about /bin/sh

ERROR - failed to execute task Command 'exec bash -c run'

I have my sysconfig file:

#!/bin/bash
PATH=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow
AIRFLOW_CONFIG=/mnt/var/airflow/airflow.cfg
AIRFLOW_HOME=/mnt/var/airflow

And my airflow-scheduler.service file:

#!/bin/bash
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service
Wants=postgresql.service

[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Here is the journalctl record which shows the bash error I am getting:

[2017-10-30 18:36:13,764] {base_executor.py:50} INFO - Adding to queue: airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/user_p
Oct 30 18:36:13  airflow[4742]: [2017-10-30 18:36:13,765] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:13  airflow[4742]: [2017-10-30 18:36:13,783] {local_executor.py:45} INFO - LocalWorker running airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/us
Oct 30 18:36:13  airflow[4742]: /bin/sh: 1: exec: bash: not found
Oct 30 18:36:13  airflow[4742]: [2017-10-30 18:36:13,865] {local_executor.py:52} **ERROR - failed to execute task Command 'exec bash -c 'airflow run** user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airf
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,786] {jobs.py:1407} INFO - Heartbeating the process manager
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,786] {dag_processing.py:559} INFO - Processor for /mnt/var/airflow/dags/bin/prod/hourly_agent_dag.py finished
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,789] {dag_processing.py:627} INFO - Started a process (PID: 5425) to generate tasks for /mnt/var/airflow/dags/bin/prod/daily_agent_email_dag.py - logging into /mnt/var/airflow/l
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,831] {jobs.py:1000} INFO - No tasks to send to the executor
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,832] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:14  airflow[4742]: [2017-10-30 18:36:14,833] {jobs.py:1195} INFO - Executor reports user_presence_raw_etl.transform_raw_user_presence execution_date=2017-10-30 14:00:00 as failed

Answer 1

Looks like your sysconfig file is clobbering the $PATH environment variable, hence the error:

Oct 30 18:36:13  airflow[4742]: /bin/sh: 1: exec: bash: not found

Try setting that line to the following:

PATH=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow:$PATH