How to see real IPs of SSH client of SSH servers running behind AWS ELB

Question

We are running two gitlab instances behind AWS classic ELB. To enable git SSH pushes and load balance SSH requests via git we added SSH TCP port listener in AWS ELB. In SSH logs we are seeing IPs of ELB not actual IPs of git users. I tried to enable Proxy protocol for SSH listener at ELB but it breaks SSH server. Is there any way to seeing real IPs of client?

Nov 16 08:38:41 gitlab-1-1b sshd[14760]: Bad protocol version identification 'PROXY TCP4 x.y.z.a 0.0.0.0 61533 22' from x.y.z.a port 9407
Nov 16 08:39:08 gitlab-1-1b sshd[14825]: Bad protocol version identification 'PROXY TCP4 x.y.z.a 0.0.0.0 61554 22' from x.y.z.a port 9417

Answer 1

It actually is possible. The solution is mmproxy which is developed by CloudFlare. It is deployed locally on each backend machine and works in front of the SSH server. When you enable the Proxy protocol in the cloud load balancer and establish a connection, "mmproxy" parses the additional header and spoofs the source IP when forwarding the connection to the local SSH server.

I've tested this with the Google Cloud TCP proxy load balancer service, but using AWS ELB should work the same way. The incoming connections were forwarded by "mmproxy" to "localhost" where OpenSSH was listening on an alternative port.

Here is my tested configuration for the Google Cloud TCP proxy load balancer service and OpenSSH:

# cat /etc/mmproxy-allowed-networks.txt
130.211.0.0/22
35.191.0.0/16

# grep Port /etc/ssh/sshd_config 
Port 222

./mmproxy -v --allowed-networks /etc/mmproxy-allowed-networks.txt -l 0.0.0.0:22 -4 127.0.0.1:222 -6 [::1]:222

Additionally, you have to deploy a custom routing table, which forces the return traffic to be routed to "loopback". For "localhost" those are four "ip" commands which must be executed every time when your machine boots.

Answer 2

As mentioned here, an ELB (Elastic Load Balancing, which does support multiple listener protocols):

acts as a forwarding proxy (i.e. the source IP is not preserved)

So all you have left is the ELB Access Logs (as mentioned there), since X-Forwarded-For, which is an "application-level" protocol, is not available for ssh natively.

So not easily possible with ssh.

---

Enabling the same GitLab service through an https listener would at least enable you to put that user IP in X-Forwarded-For through proxied SSL support, as discussed in this GitLab thread, or this one.
Note that recent GitLab (8.10 or more) would then require

proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Port 443;