Menu
Is it possible to customize the format that Airflow uses for logging?
I tried adding a LOG_FORMAT variable in $AIRFLOW_HOME/airflow.cfg, but it doesn't seem to take effect
LOG_FORMAT = "%(asctime)s logLevel=%(levelname)s logger=%(name)s - %(message)s"
922
Airflow writes logs for tasks in a way that allows to see the logs for each task separately via Airflow UI. The Core Airflow implements writing and serving logs locally. However you can also write logs to remote services - via community providers, but you can also write your own loggers.
Apache Airflow is a data pipeline orchestration tool. It helps run periodic jobs that are written in Python, monitor their progress and outcome, retry failed jobs and convey events in a colourful and concise Web UI. Jobs, known as DAGs, have one or more tasks.
This can be done by logging_config_class option in airflow.cfg file. This option should specify the import path indicating to a configuration compatible with logging.config.dictConfig (). If your file is a standard import location, then you should set a PYTHONPATH environment. Follow the steps below to enable custom logging config class:
You can also choose to have the logs output in a JSON format, using the json_format option. Airflow uses the standard Python logging module and JSON fields are directly extracted from the LogRecord object. To use this feature, set the json_fields option in airflow.cfg.
You need to change the settings.py file in the airflow package to change the log format
Update settings.py (after LOGGING_LEVEL add below line):
LOG_FORMAT = os.path.expanduser(conf.get('core', 'LOG_FORMAT'))
Update airflow.cfg configuration file: Add line under [core]:
LOG_FORMAT = "%(asctime)s logLevel=%(levelname)s logger=%(name)s - %(message)s"
Restart webserver and scheduler services
Try the approach as mentioned in the answer here to change logging level
83
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
