Cannot ssh to remote hosts on any port other than 22 from macOS using built-in ssh client

Question

I recently switched the ssh listen port on my server running Dropbear from 22 to a random one to prevent the system log from being flooded by someone brute-forcing.

Everything is fine and I am able to connect to the server from the wan side using termux(an Android terminal emulator that you can install packages).

Until when I try using my MacBook to ssh into the server(under the same network with my phone, was previously able to ssh into the server when dropbear was listening on 22). The connection immediately drops and ssh throws this at me:

kex_exchange_identification: write: Broken pipe

The verbose output does not really show anything helpful(and cut really abruptly may I add):

OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/my_username/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to remote.yhaoquan.top port 2123.
debug1: Connection established.
debug1: identity file /Users/my_username/.ssh/id_rsa type 0
debug1: identity file /Users/my_username/.ssh/id_rsa-cert type -1
debug1: identity file /Users/my_username/.ssh/id_dsa type -1
debug1: identity file /Users/my_username/.ssh/id_dsa-cert type -1
debug1: identity file /Users/my_username/.ssh/id_ecdsa type -1
debug1: identity file /Users/my_username/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/my_username/.ssh/id_ed25519 type -1
debug1: identity file /Users/my_username/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/my_username/.ssh/id_xmss type -1
debug1: identity file /Users/my_username/.ssh/id_xmss-cert type -1
kex_exchange_identification: write: Broken pipe

Telnetting to port 2123 shows that I actually successfully made the connection to the server:

>telnet {hostname} 2123
Trying {host IP address}...
Connected to {hostname}
Escape character is '^]'.
SSH-2.0-dropbear
|

��]J044��d����curve25519-sha256,[email protected],diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,[email protected],[email protected],aes128-ctr,[email protected],aes128-ctr,aes256-ctrhmac-sha1,hmac-sha2-256hmac-sha1,hmac-sha2-256nonenoneM�'�Rf

When I change the port back to 22 on my server, everything works again.

Does anyone know where the problem might be or what way should I go about diagnosing the problem? Thanks in advance :)

EDIT 1: For those of you who are experiencing the same issue, a temporary solution would be using OpenSSH. The easiest way (that makes this a drop-in solution) to do this is by using Homebrew:

brew install openssh

This command will change the ssh client you use from /usr/bin/ssh to an alias placed at /usr/local/bin/ssh points to the OpenSSH you just installed using brew, which the symptom doesn't seem to appear on.

This does not solve the problem that this post is about. I am still looking for a solution for it.

Answer 1

I had the same issue: ssh returns "kex_exchange_identification: write: Broken pipe" without network activity.

In my case the guilty party was AdGuard.

I added the port I use for ssh in the advanced configuration, network.extension.exclude.ports and it works now. network.extension.exclude.domains may be a better choice for you depending on your configuration.

Answer 2

This solved my issue with /usr/bin/ssh.

Background:

• ssh to a named host would fail with kex_exchange_identification: write: Broken pipe.
• ssh to an IP address would work.
• Installing a new version of SSH (using Homebrew) worked (from the terminal, using /usr/local/bin/ssh).
• Mac Mail was still broken (it uses SSH to connect to my IMAP accounts).

I found this post where someone (yesterday) was experiencing the exact same symptoms as me from a piece of Broadcom antivirus software (WSS Agent) which I do not use:

https://community.broadcom.com/symantecenterprise/communities/community-home/digestviewer/viewthread?GroupId=2593&MessageKey=57c16f08-06e4-4c1d-adc7-bd3f51cc4516&CommunityKey=84278e0f-9c57-4f93-8cff-4530b03e3c07&tab=digestviewer&ReturnUrl=%2Fbrowse%2Fallrecentposts

That reminded me that Sophos (Antivirus) had recently (maybe yesterday, maybe the day before) installed a "SophosWebNetworkExtentsion". They posted this video (of how to accept the install) on 2021 Jan 19th (two days ago):

https://www.youtube.com/watch?v=z-qWdv1ynMs

The Fix:

1. I logged into Sophos, and under "Web Protection" disabled Web Protection.
2. I needed to reboot for the change to affect SSH. After rebooting, usr/bin/ssh started working again and Mac Mail started working again.

Before the fix:

$ /usr/bin/ssh s0-backend -v
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/eric/.ssh/config
debug1: /Users/eric/.ssh/config line 62: Applying options for s0-backend
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to s0-backend port 61917.
debug1: Connection established.
debug1: identity file /Users/eric/.ssh/id_rsa type 0
debug1: identity file /Users/eric/.ssh/id_rsa-cert type -1
debug1: identity file /Users/eric/.ssh/id_dsa type -1
debug1: identity file /Users/eric/.ssh/id_dsa-cert type -1
debug1: identity file /Users/eric/.ssh/id_ecdsa type -1
debug1: identity file /Users/eric/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/eric/.ssh/id_ed25519 type -1
debug1: identity file /Users/eric/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/eric/.ssh/id_xmss type -1
debug1: identity file /Users/eric/.ssh/id_xmss-cert type -1
kex_exchange_identification: write: Broken pipe
$

After the fix (successful SSH login):

usr/bin/ssh s0-backend -v
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/eric/.ssh/config
debug1: /Users/eric/.ssh/config line 62: Applying options for s0-backend
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to s0-backend port {redacted}.
debug1: Connection established.
debug1: identity file /Users/eric/.ssh/id_rsa type 0
debug1: identity file /Users/eric/.ssh/id_rsa-cert type -1
debug1: identity file /Users/eric/.ssh/id_dsa type -1
debug1: identity file /Users/eric/.ssh/id_dsa-cert type -1
debug1: identity file /Users/eric/.ssh/id_ecdsa type -1
debug1: identity file /Users/eric/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/eric/.ssh/id_ed25519 type -1
debug1: identity file /Users/eric/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/eric/.ssh/id_xmss type -1
debug1: identity file /Users/eric/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to s0-backend:61917 as '{redacted}'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-ed25519 SHA256: {redacted}
debug1: Host '[s0-backend]:{redacted}' is known and matches the ED25519 host key.
debug1: Found key in /Users/eric/.ssh/known_hosts:40
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
...
{redacted}
...
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_TERMINAL = iTerm2
debug1: Sending env LC_TERMINAL_VERSION = 3.4.3
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-62-generic x86_64)
...
{redacted}
...
$