Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to integrate Apache Airflow with slack?

Tags:

slack

airflow

could someone please give me step by step manual on how to connect Apache Airflow to Slack workspace. I created webhook to my channel and what should I do with it next ?

Kind regards

like image 360
Clyde Barrow Avatar asked Aug 28 '18 09:08

Clyde Barrow


People also ask

What is Apache Airflow and slack?

Slack is an increasingly popular chat app used in the workplace. Apache Airflow is an open source platform for orchestrating workflows. One of the biggest advantages to using Airflow is the versatility around its hooks and operators.

What are slack hooks and operators?

Slack is an increasingly popular chat app used in the workplace. Apache Airflow is an open source platform for orchestrating workflows. One of the biggest advantages to using Airflow is the versatility around its hooks and operators. Hooks are interfaces to external platforms, databases and also serve as the basic building blocks of Operators.

What is a slack webhook?

Hooks are interfaces to external platforms, databases and also serve as the basic building blocks of Operators. The Slack Webhook Operator ca n be used to integrate Airflow with Slack. This operator is typically used for reporting and alerting purposes by scheduling incoming messages to Slack channels when some trigger condition is met.

Should you monitor your airflow pipelines via slack?

If you are already using Slack as a messenger, then monitoring via Slack is a great option instead of using yet another tool that you and your team need to check regularly. Furthermore, since everybody already knows how to work with Slack, there is no entry barrier. Monitoring via Slack works nicely with a few Airflow pipelines or a hundred.


1 Answers

  • Create a Slack Token from https://api.slack.com/custom-integrations/legacy-tokens
  • Use the SlackAPIPostOperator Operator in your DAG as below
SlackAPIPostOperator(
      task_id='failure',
      token='YOUR_TOKEN',
      text=text_message,
      channel=SLACK_CHANNEL,
      username=SLACK_USER)

The above is the simplest way you can use Airflow to send messages to Slack.

However, if you want to configure Airflow to send messages to Slack on task failures, create a function and add on_failure_callback to your tasks with the name of the created slack function. An example is below:

def slack_failed_task(contextDictionary, **kwargs):  
       failed_alert = SlackAPIPostOperator(
         task_id='slack_failed',
         channel="#datalabs",
         token="...",
         text = ':red_circle: DAG Failed',
         owner = '_owner',)
         return failed_alert.execute()


task_with_failed_slack_alerts = PythonOperator(
    task_id='task0',
    python_callable=<file to execute>,
    on_failure_callback=slack_failed_task,
    provide_context=True,
    dag=dag)

Using SlackWebHook (Works only for Airflow >= 1.10.0): If you want to use SlackWebHook use SlackWebhookOperator in a similar manner:

https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/slack_webhook_operator.py#L25

like image 192
kaxil Avatar answered Sep 28 '22 11:09

kaxil