Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I add a new connection type to Airflow?

Tags:

airflow

Looking at the plugin documentation (https://airflow.incubator.apache.org/plugins.html), it is clear how to add to new hooks and operators, but I am adding a new hook that requires connection information. This information seems to be hard-coded in airflow/models.py. Is there a way to add my own connection type to the list without altering Airflow's source code?

like image 919
Nick Avatar asked Feb 14 '18 13:02

Nick


People also ask

How do you add a snowflake connection to Airflow?

Configuring the ConnectionSpecify the snowflake password. For public key authentication, the passphrase for the private key. Specify the snowflake schema to be used. Specify the extra parameters (as json dictionary) that can be used in the snowflake connection.

How do I connect to Airflow in MySQL?

Configuring the ConnectionSpecify the user name to connect. Specify the password to connect. Specify the extra parameters (as json dictionary) that can be used in MySQL connection. Note that you can choose the client to connect to the database by setting the client extra field.


1 Answers

airflow Connection's conn_type field allows null value. so if you don't care about giving unique type name to your custom hook, then you can give a default connection value in your hook implementation.

from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
from airflow.utils.db import provide_session 

class MyHook(BaseHook):

    # ... impl whatever you want.        

    @classmethod
    @provide_session
    def get_hook(cls, conn_id='myhook_default', session=None):
        try:
            conn = cls.get_connection(conn_id)
        except AirflowException:  
            # create default connection. run only once.
            conn = Connection(
                conn_id=conn_id,
                # conn_type='string500',  # You can give new type string here. But no UI component's for you. Just leave it.
                host='default.example.com',
                port=80,
                login='default_login',
                password='default_pw',
                extra=json.dumps({...extra_defult_you_need...}),
            )
            session.add(conn)
            session.commit()
    return MyHook(conn=conn)
like image 102
Han-Jong Ko Avatar answered Sep 20 '22 09:09

Han-Jong Ko