Menu
I am using Airflow 1.7.1.3 installed using pip
I would like to limit the logging to ERROR level for the workflow being executed by the scheduler. Could not find anything beyond setting log files location in the settings.py file.
Also the online resources led me to this google group discussion here but not much info here as well
Any idea how to control logging in Airflow?
369
For example, if you wanted to change your logging level from the default INFO to only log LogRecords with a level of ERROR or above, you could set logging_level = ERROR in airflow. cfg or define an environment variable AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR .
You can also view the logs in the Airflow web interface. Streaming logs: These logs are a superset of the logs in Airflow. To access streaming logs, you can go to the logs tab of Environment details page in Google Cloud console, use the Cloud Logging, or use Cloud Monitoring. Logging and Monitoring quotas apply.
The first time you run Airflow, it will create a file called airflow. cfg in your $AIRFLOW_HOME directory ( ~/airflow by default). This file contains Airflow's configuration and you can edit it to change any of the settings.
Airflow writes logs for tasks in a way that allows to see the logs for each task separately via Airflow UI. The Core Airflow implements writing and serving logs locally. However you can also write logs to remote services - via community providers, but you can also write your own loggers.
To change the log level for the Apache webserver, we have to add the below line of code in the Apache webserver configuration file: In the above line, we have set the log level to warning. Below we have a sample Apache configuration file, where we have also used the ErrorLog configuration to change the log file location too, to a custom location.
By default, logs are placed in the AIRFLOW_HOME directory. The following convention is followed while naming logs: {dag_id}/ {task_id}/ {logical_date}/ {try_number}.log In addition, users can supply a remote location to store current logs and backups. In the Airflow Web UI, remote logs take precedence over local logs when remote logging is enabled.
See Modules Management for details on how Python and Airflow manage modules. Most task handlers send logs upon completion of a task. In order to view logs in real time, airflow automatically starts an http server to serve the logs in the following cases: If SchedulerExecutor or LocalExecutor is used, then when airflow scheduler is running.
I tried below work around and it seems to be working to set LOGGING_LEVEL outside of settings.py:
Update settings.py:
Remove or comment line:LOGGING_LEVEL = logging.INFO
Add line:LOGGING_LEVEL = os.path.expanduser(conf.get('core', 'LOGGING_LEVEL'))
Update airflow.cfg configuration file:
Add line under [core]:
logging_level = WARN
Restart webserver and scheduler services
Use environment vaiable AIRFLOW__CORE__LOGGING_LEVEL=WARN.
See the official docs for details.
121
The logging functionality and its configuration will be changed in version 1.9 with this commit
38
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
