In Gitlab CI build, I can't ssh into AWS EC2 by private key

Question

At first, I was trying to use ansible for deployment after gitlab CI built, but it shows "host unreachable" in result.

After some trial and error, I found out the problem is ssh permission denied when ssh by private key into my AWS EC2 instance to deploy.

My .gitlab-ci.yml config is something like this:

.gitlab-ci.yml

image: ansible/ubuntu14.04-ansible:stable

stages:
  - deploy

deploy_web:
  stage: deploy
  script:
   - "echo Ansible"
   - "echo Environment: ${ENV}"
   - "echo TAG: ${TAG}"

   - "echo ${VAULT_PASS} > vault_pass.txt"
   - "mkdir sshkey"
   - "echo ${SSH_KEY_APP} > ./sshkey/app-key.pem"
   - "chmod 600 ./sshkey/app-key.pem"
   - "export SSH_KEY_DIR=`pwd`/sshkey"
   - "export ANSIBLE_HOST_KEY_CHECKING=False"
   - "ssh-keyscan foobar.io >> ~/.ssh/known_hosts"
   - "ssh -v -i ./sshkey/app-key.pem [email protected]" // for debugging

   - "ansible-playbook -i ${ENV} servers.yml --vault-password-file vault_pass.txt -vvvv --tags=${TAG}"

When gitlab CI builds this, it basically gives these ssh error messages:

OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to foobar.io [12.34.56.78] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file ./sshkey/app-key.pem type -1
debug1: identity file ./sshkey/app-key.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr [email protected] none
debug1: kex: client->server aes128-ctr [email protected] none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA be:b1:53:76:aa:bf:65:ea:b4:1b:7a:8f:cc:7c:2a:79
debug1: Host 'foobar.io' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
Warning: Permanently added the ECDSA host key for IP address '12.34.56.78' to the list of known hosts.
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: ./sshkey/app-key.pem
debug1: key_parse_private2: missing begin marker
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: No more authentication methods to try.

Permission denied (publickey).

Also tried using absolute path:

$ cat /builds/foobar/bar/sshkey/app-key.pem
-----BEGIN RSA PRIVATE KEY-----
 ...(the key)...
 -----END RSA PRIVATE KEY-----

$ ssh -v -i /builds/foobar/bar/sshkey/app-key.pem [email protected]
Permission denied (publickey).

These are what I have tried:

• try using shell executor for gitlab CI runner -> failed
• run the scripts in a local docker container -> success
• ssh into the runner instance manually (not through CI) and run the scripts in shell -> success
• ssh into the runner instance manually and run the scripts in docker container -> success

As a conclusion - It only fails when run by gitlab CI, so I wonder if there are any additional configuration I haven't noticed to do things like this...

Many thanks for anyone can help!

The real problem is

When echo-ing a multiline environment variable, quotes are needed. So basically every line of the key ends with ^M, which shows correctly in gitlab's console but actually unable to be parsed by ssh.

Answer 1

If it fails when run by GilabCI, it means the user used by GitLab CI is not the same as the one used when you ssh in the running instance.

See for instance "AWS SSH connection error: Permission denied (publickey)"

Another thing to check is PermitRootLogin and AllowUsers in /etc/ssh/sshd_config.

This debug1: key_parse_private2: missing begin marker appears even after successful key authorization if your user access restricted.

Check after a manually ssh on the remote machine:

tail -f -n 80 /var/log/auth.log

The OP DarkBtf adds in the comments:

When echo-ing a multiline environment variable, quotes are needed.
So basically every line of the key ends with ^M, which shows correctly in gitlab's console but actually unable to be parsed by ssh.