Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Airflow: How to SSH and run BashOperator from a different server

Is there a way to ssh to different server and run BashOperator using Airbnb's Airflow? I am trying to run a hive sql command with Airflow but I need to SSH to a different box in order to run the hive shell. My tasks should look like this:

  1. SSH to server1
  2. start Hive shell
  3. run Hive command

Thanks!

like image 673
CMPE Avatar asked Sep 12 '16 19:09

CMPE


People also ask

How do you use a SSHOperator in airflow?

SSHOperator to execute commands on given remote host using the ssh_hook. ssh_hook (SSHHook | None) – predefined ssh_hook to use for remote execution. Either ssh_hook or ssh_conn_id needs to be provided. ssh_conn_id (str | None) – ssh connection id from airflow Connections.

What is BashOperator in airflow?

The BashOperator is one of the most commonly used operators in Airflow. It executes bash commands or a bash script from within your Airflow DAG.


1 Answers

NOT available for airflow 2.x.

I think that I just figured it out:

  1. Create a SSH connection in UI under Admin > Connection. Note: the connection will be deleted if you reset the database

  2. In the Python file add the following

     from airflow.contrib.hooks import SSHHook  sshHook = SSHHook(conn_id=<YOUR CONNECTION ID FROM THE UI>) 
  3. Add the SSH operator task

     t1 = SSHExecuteOperator(      task_id="task1",      bash_command=<YOUR COMMAND>,      ssh_hook=sshHook,      dag=dag) 

Thanks!

like image 101
CMPE Avatar answered Oct 04 '22 04:10

CMPE