I got the webserver to work properly, and my airflow-scheduler.service file starts the scheduler and it finds my dags etc. However, tasks are not running:
I see an error message about /bin/sh
ERROR - failed to execute task Command 'exec bash -c run'
I have my sysconfig file:
#!/bin/bash
PATH=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow
AIRFLOW_CONFIG=/mnt/var/airflow/airflow.cfg
AIRFLOW_HOME=/mnt/var/airflow
And my airflow-scheduler.service file:
#!/bin/bash
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service
Wants=postgresql.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
Here is the journalctl record which shows the bash error I am getting:
[2017-10-30 18:36:13,764] {base_executor.py:50} INFO - Adding to queue: airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/user_p
Oct 30 18:36:13 airflow[4742]: [2017-10-30 18:36:13,765] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:13 airflow[4742]: [2017-10-30 18:36:13,783] {local_executor.py:45} INFO - LocalWorker running airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/us
Oct 30 18:36:13 airflow[4742]: /bin/sh: 1: exec: bash: not found
Oct 30 18:36:13 airflow[4742]: [2017-10-30 18:36:13,865] {local_executor.py:52} **ERROR - failed to execute task Command 'exec bash -c 'airflow run** user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airf
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,786] {jobs.py:1407} INFO - Heartbeating the process manager
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,786] {dag_processing.py:559} INFO - Processor for /mnt/var/airflow/dags/bin/prod/hourly_agent_dag.py finished
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,789] {dag_processing.py:627} INFO - Started a process (PID: 5425) to generate tasks for /mnt/var/airflow/dags/bin/prod/daily_agent_email_dag.py - logging into /mnt/var/airflow/l
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,831] {jobs.py:1000} INFO - No tasks to send to the executor
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,832] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,833] {jobs.py:1195} INFO - Executor reports user_presence_raw_etl.transform_raw_user_presence execution_date=2017-10-30 14:00:00 as failed
We first need to download the service definition files from the Apache Airflow GitHub repo, then put them into the correct system directories. We also need to create some folders because the daemon will need them to run correctly.
If you run Airflow locally and start it with the two commands airflow scheduler and airflow webserver , then those processes will run in the foreground. So, simply hitting Ctrl-C for each of them should terminate them and all their child processes.
cfg in your $AIRFLOW_HOME directory ( ~/airflow by default). This file contains Airflow's configuration and you can edit it to change any of the settings. You can also set options with environment variables by using this format: AIRFLOW__{SECTION}__{KEY} (note the double underscores).
Looks like your sysconfig file is clobbering the $PATH
environment variable, hence the error:
Oct 30 18:36:13 airflow[4742]: /bin/sh: 1: exec: bash: not found
Try setting that line to the following:
PATH=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow:$PATH
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With