SSH Host Key Verification Failed inside GitLab CI

Question

Local Setup

I created a public and private SSH key via the ssh-keygen command.

I decided to setup the private key locally first, before setting it up on my repo's gitlab CI.

I setup the public key on the server (in this case, another gitlab repo, but this may change in the future and shouldn't affect the question).

I successfully communicated with the server locally via the following command (in this case I am using SSH via git, but this again may change in the future):

git clone [email protected]:...../......git

GitLab CI Setup

I then decided to setup the private key and communication on gitlab CI.

Inside my repo, I navigated to Settings -> Continuous Integration -> Variables, and added the following environment variables:

• SSH_DEPLOY_PRIVATE_KEY - I used to same private key that I used locally
• SSH_KNOWN_HOSTS
  • I took the gitlab.com known host from my local computer's ~/.ssh/known_hosts file
  • gitlab.com,35.231.145.151 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=

I then setup the SSH inside .gitlab-ci.yml:

script:
  - apt-get install openssh-client -y
  - eval $(ssh-agent -s)
  - echo "$SSH_DEPLOY_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
  - mkdir -p /.ssh && touch /.ssh/known_hosts
  - echo "$SSH_KNOWN_HOSTS" >> /.ssh/known_hosts
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

This seemed to work fine and I got the following message: Identity added: (stdin) (runner@....)

I then added the same git clone command to communicate with the server, and it failed with the following error:

Cloning into '......'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Testing locally still works. I used the same commands above to setup the SSH locally (except I used pacman -S openssh to install instead).

How do I fix this?

Edit

I'm aware that I can execute ssh-keyscan directly in the GitLab CI and this should in theory solve the problem, but from what I know, this is susceptible to man-in-the-middle attacks. I'm trying to find a more secure solution.

Edit 2

After running ssh-keyscan directly in the GitLab CI, I get the same error message.

Verbose output is the same:

$ GIT_SSH_COMMAND="ssh -vvv" git clone [email protected]:..../.....git deployed
Cloning into 'deployed'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Edit 3

Seems to be connected to the internet. Plus apt-get install wouldn't work otherwise.

Edit 4

I don't understand why this is such a difficult task. I followed this article and am doing everything correctly. There seems to be plenty of other similar questions, which also do not have any answers. Is this just an issue with GitLab CI that we have no control over?

I'm also now thinking that it has something to do with the fact that the SSH server is another GitLab repo. Maybe GitLab CI blocks SSH connections within the same network. Not sure why but it's a possibility. Also don't know how you'd connect without SSH.

Edit 5

The verbose output clearly was not working using GIT_SSH_COMMAND, so I tried an ssh connection without git:

ssh -vvvv [email protected]

Log output:

OpenSSH_6.7p1 Debian-5+deb8u5, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug2: ssh_connect: needpriv 0
debug1: Connecting to gitlab.com [35.231.145.151] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "gitlab.com" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: setup [email protected]
debug1: kex: server->client aes128-ctr [email protected] none
debug2: mac_setup: setup [email protected]
debug1: kex: client->server aes128-ctr [email protected] none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA f1:d0:fb:46:73:7a:70:92:5a:ab:5d:ef:43:e2:1c:35
debug3: load_hostkeys: loading entries for host "gitlab.com" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug3: load_hostkeys: loading entries for host "35.231.145.151" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: read_passphrase: can't open /dev/tty: No such device or address
Host key verification failed.

The second-last line indicates that it's trying to communicate with the terminal using the /dev/tty file. Of course, this script is running in a non-interactive manor so it fails. Shouldn't it be using my key instead of requesting a passphrase from the terminal?

Answer 1

You may need to try setting the mode to 644 rather than 700. 644 is what is suggested in the Verifying the SSH host keys documentation, and is also what SSH uses for this file by default. Some parts of SSH are very particular about this - I'm not sure if known_hosts is particular.

The docs also mention you should set the value of SSH_KNOWN_HOSTS variable to the entire output of ssh-keyscan since there are multiple keys.

EDIT:

The following .gitlab-ci.yml worked for me on GitLab.com. Note the use of ~/.ssh/ rather than /.ssh/.

image: ubuntu:latest

test_job:
  script:
  - apt-get update
  - apt-get install openssh-client git-core -y
  - eval $(ssh-agent -s)
  - echo "$SSH_DEPLOY_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
  - mkdir -p ~/.ssh && touch ~/.ssh/known_hosts
  - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
  - git clone [email protected]:gitlab-org/gitlab-ce.git

Answer 2

A server can have multiple signatures in case of load balanced server. Ideal way is to change

- echo "$SSH_KNOWN_HOSTS" >> /.ssh/known_hosts

to

- ssh-keyscan www.gitlab.com >> /.ssh/known_hosts