Is there a way to ssh to different server and run BashOperator using Airbnb's Airflow? I am trying to run a hive sql command with Airflow but I need to SSH to a different box in order to run the hive shell. My tasks should look like this:
Thanks!
SSHOperator to execute commands on given remote host using the ssh_hook. ssh_hook (SSHHook | None) – predefined ssh_hook to use for remote execution. Either ssh_hook or ssh_conn_id needs to be provided. ssh_conn_id (str | None) – ssh connection id from airflow Connections.
The BashOperator is one of the most commonly used operators in Airflow. It executes bash commands or a bash script from within your Airflow DAG.
I think that I just figured it out:
Create a SSH connection in UI under Admin > Connection. Note: the connection will be deleted if you reset the database
In the Python file add the following
from airflow.contrib.hooks import SSHHook sshHook = SSHHook(conn_id=<YOUR CONNECTION ID FROM THE UI>)
Add the SSH operator task
t1 = SSHExecuteOperator( task_id="task1", bash_command=<YOUR COMMAND>, ssh_hook=sshHook, dag=dag)
Thanks!
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With