Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there documentation for Airflow log file return codes?

I am trying to identify what in a .log file makes a specific task marked as "Successful" vs "Failure" in airflow. I am getting the sense that this is dependent upon the operator (PythonOperator vs EmailOperator vs BashOperator etc.).

I am seeing different return codes. For example, INFO - Task exited with return code 1 or INFO - Task exited with return code 0 or INFO - Process psutil.Process(pid=00000, status='terminated') (00000) terminated with exit code -15

I am not seeing consistency.. I first thought INFO - Task exited with return code 0 constituted a success, but I see some failure logs also have this.

I would appreciate any insight into understanding different return code #.

like image 458
Josh Avatar asked Jul 25 '19 14:07

Josh


People also ask

How do you read Airflow logs?

You can also view the logs in the Airflow web interface. Streaming logs: These logs are a superset of the logs in Airflow. To access streaming logs, you can go to the logs tab of Environment details page in Google Cloud console, use the Cloud Logging, or use Cloud Monitoring. Logging and Monitoring quotas apply.

How do you add logs to Airflow?

You can just import logging in Python and then do logging.info('whatever logs you want') and that will write to the Airflow logs. You don't need to invoke your Python code through the BashOperator just use the PythonOperator.

How does Airflow detect that a Python file is a DAG?

Airflow loads DAGs from Python source files, which it looks for inside its configured DAG_FOLDER . It will take each file, execute it, and then load any DAG objects from that file. This means you can define multiple DAGs per Python file, or even spread one very complex DAG across multiple Python files using imports.


1 Answers

It seems that these return codes are equal that exit codes from python. The meaning of these exit codes are better explained in this link, but in summary, code 0 means "no errors" in your process, and 1 means that your process has one or more errors.

like image 170
Leonardo Gasparini Avatar answered Sep 28 '22 20:09

Leonardo Gasparini