Bash Operator error: No such file or directory in airflow

Question

I am a newbie to Airflow and struggling with BashOperator. I want to access a shell script using bash operatory in my dag.py.

I checked: How to run bash script file in Airflow and BashOperator doen't run bash file apache airflow

on how to access shell script through bash operator.

This is what I did:

 cmd = "./myfirstdag/dag/lib/script.sh "

        t_1 = BashOperator(
            task_id='start',
            bash_command=cmd
        )

On running my recipe and checking in airflow I got the below error:

[2018-11-01 10:44:05,078] {bash_operator.py:77} INFO - /tmp/airflowtmp7VmPci/startUDmFWW: line 1: ./myfirstdag/dag/lib/script.sh: No such file or directory
[2018-11-01 10:44:05,082] {bash_operator.py:80} INFO - Command exited with return code 127
[2018-11-01 10:44:05,083] {models.py:1361} ERROR - Bash command failed

Not sure why this is happening. Any help would be appreciated.

Thanks !

EDIT NOTE: I assume that it's searching in some airflow tmp location rather than the path I provided. But how do I make it search for the right path.

Answer 1

Try this:

bash_operator = BashOperator(
    task_id = 'task',
    bash_command = '${AIRFLOW_HOME}/myfirstdag/dag/lib/script.sh '
    dag = your_dag)

Answer 2

For those running a docker version.

I had this same issue, took me a while to realise the problem, the behaviour can be different with docker. When the DAG is run it moves it tmp file, if you do not have airflow on docker this is on the same machine. with my the docker version it moves it to another container to run, which of course when it is run would not have the script file on.

check the task logs carefully, you show see this happen before the task is run. This may also depend on your airflow-docker setup.