Menu
I am trying to identify what in a .log file makes a specific task marked as "Successful" vs "Failure" in airflow. I am getting the sense that this is dependent upon the operator (PythonOperator vs EmailOperator vs BashOperator etc.).
I am seeing different return codes. For example, INFO - Task exited with return code 1 or INFO - Task exited with return code 0 or INFO - Process psutil.Process(pid=00000, status='terminated') (00000) terminated with exit code -15
I am not seeing consistency.. I first thought INFO - Task exited with return code 0 constituted a success, but I see some failure logs also have this.
I would appreciate any insight into understanding different return code #.
458
You can also view the logs in the Airflow web interface. Streaming logs: These logs are a superset of the logs in Airflow. To access streaming logs, you can go to the logs tab of Environment details page in Google Cloud console, use the Cloud Logging, or use Cloud Monitoring. Logging and Monitoring quotas apply.
You can just import logging in Python and then do logging.info('whatever logs you want') and that will write to the Airflow logs. You don't need to invoke your Python code through the BashOperator just use the PythonOperator.
Airflow loads DAGs from Python source files, which it looks for inside its configured DAG_FOLDER . It will take each file, execute it, and then load any DAG objects from that file. This means you can define multiple DAGs per Python file, or even spread one very complex DAG across multiple Python files using imports.
It seems that these return codes are equal that exit codes from python. The meaning of these exit codes are better explained in this link, but in summary, code 0 means "no errors" in your process, and 1 means that your process has one or more errors.
170
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
