ERROR - SSH operator error: timed out in Airflow while connecting with ec2 instance [closed]

Question

I am getting a timeout error while trying to connect with ec2 instance. I can connect to the machine through ssh command:

ssh -i keypair.pem myuser@ec2IPaddress

My connection file has the following things:

conn Id: ssh_custom Conn Type: SSH Host: ec2IPaddress Username: myuser Port: 8888 Extra: {"key_file":"/home/ubuntu/keypair.pem", "no_host_key_check": "false", "allow_host_key_change": "true", "timeout": "45"}

My dag file looks like:

from airflow import DAG
import datetime as dt
from airflow.operators.bash_operator import BashOperator
from airflow.contrib.hooks.ssh_hook import SSHHook
from airflow.contrib.operators.ssh_operator import SSHOperator

sshHook = SSHHook(ssh_conn_id='ssh_custom')

default_args = {
    'owner': 'arpita',
    'start_date': dt.datetime(2019, 11, 20),
    'retries': 1,
    'retry_delay': dt.timedelta(minutes=5),
    'depends_on_past': False,
    'email': ['[email protected]'],
    'email_on_failure': True,
    'email_on_retry': True,
}

with DAG('sample',
         default_args=default_args,
         schedule_interval='30 * * * *',
         ) as dag:
     task1 = SSHOperator(task_id="task1",
                         command='echo $HOSTNAME',
                         ssh_hook=sshHook)

task1

Getting the error:

ERROR - SSH operator error: timed out
Traceback (most recent call last):
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 101, in execute
    with self.ssh_hook.get_conn() as ssh_client:
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/hooks/ssh_hook.py", line 180, in get_conn
    sock=self.host_proxy)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/util.py", line 280, in retry_on_signal
    return function()
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 926, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 173, in execute
    raise AirflowException("SSH operator error: {0}".format(str(e)))
airflow.exceptions.AirflowException: SSH operator error: timed out
[2019-11-26 15:23:20,254] {taskinstance.py:1080} INFO - All retries failed; marking task as FAILED
[2019-11-26 15:23:20,284] {logging_mixin.py:95} INFO - [[34m2019-11-26 15:23:20,284[0m] {[34mconfiguration.py:[0m299} WARNING[0m - section/key [[1msmtp[0m/[1msmtp_user[0m] not found in config[0m
[2019-11-26 15:23:20,284] {taskinstance.py:1086} ERROR - Failed to send email to: ['[email protected]']
[2019-11-26 15:23:20,285] {taskinstance.py:1087} ERROR - [Errno 111] Connection refused
Traceback (most recent call last):
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 101, in execute
    with self.ssh_hook.get_conn() as ssh_client:
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/hooks/ssh_hook.py", line 180, in get_conn
    sock=self.host_proxy)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/util.py", line 280, in retry_on_signal
    return function()
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/paramiko/client.py", line 349, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 926, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/contrib/operators/ssh_operator.py", line 173, in execute
    raise AirflowException("SSH operator error: {0}".format(str(e)))
airflow.exceptions.AirflowException: SSH operator error: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 1084, in handle_failure
    self.email_alert(error)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/models/taskinstance.py", line 1307, in email_alert
    send_email(self.task.email, subject, html_content)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/utils/email.py", line 55, in send_email
    mime_subtype=mime_subtype, mime_charset=mime_charset, **kwargs)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/utils/email.py", line 101, in send_email_smtp
    send_MIME_email(smtp_mail_from, recipients, msg, dryrun)
  File "/home/ubuntu/airflow/.venv/lib/python3.5/site-packages/airflow/utils/email.py", line 121, in send_MIME_email
    s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT)
  File "/usr/lib/python3.5/smtplib.py", line 251, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib/python3.5/smtplib.py", line 335, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib/python3.5/smtplib.py", line 306, in _get_socket
    self.source_address)
  File "/usr/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/usr/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Answer 1

The problem is solved, actually, the port number 8888 was incorrect. It worked with port number 22.