How to run bash script file in Airflow

Question

I have a bash script that creates a file (if it does not exist) that I want to run in Airflow, but when I try it fails. How do I do this?

#!/bin/bash #create_file.sh  file=filename.txt  if [ ! -e "$file" ] ; then     touch "$file" fi  if [ ! -w "$file" ] ; then     echo cannot write to $file     exit 1 fi 

and here's how I'm calling it in Airflow:

create_command = """  ./scripts/create_file.sh """ t1 = BashOperator(         task_id= 'create_file',         bash_command=create_command,         dag=dag )  lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 83, in execute     raise AirflowException("Bash command failed") airflow.exceptions.AirflowException: Bash command failed 

Answer 1

From the tutorial this is OK:

t2 = BashOperator(     task_id='sleep',     bash_command='sleep 5',     retries=3,     dag=dag) 

But you're passing a multi-line command to it

create_command = """  ./scripts/create_file.sh """ 

should be

create_command = "./scripts/create_file.sh " 

Moreover, you also have to make sure that you are in the correct directory to avoid cryptic errors. Do it like this for example:

create_command = "./scripts/create_file.sh " if os.path.exists(create_command):    t1 = BashOperator(         task_id= 'create_file',         bash_command=create_command,         dag=dag    ) else:     raise Exception("Cannot locate {}".format(create_command)) 

Answer 2

From the documentation:

t2 = BashOperator(     task_id='bash_example',     # "scripts" folder is under "/usr/local/airflow/dags"     bash_command="scripts/test.sh",     dag=dag)