GitLab SSH keys stopped working

Question

OK, I've stumbled on this topic many times, but this is the first time that none of the regular solutions worked.

I have a CentOS 6.4 server running GitLab. It's been working great with 20+ users and 60+ projects, but about 5 hours ago, my main staging server was unable to connect to the GitLab machine for the first time using key authentication and it prompted for password. I regenerated the RSA key and added it to my deploy keys, but that failed as well.

Next, I tried to create a new user on the staging server, create a key for it, and add it to GitLab but it still fails.

Permissions:

drwxr-x---  22 root root 4.0K Oct 28 13:20 root

Inside root:

drwx------   2 root root     4096 Oct 28 11:49 .ssh

Inside .ssh:

-rw-------  1 root root  227 Oct 28 11:48 authorized_keys
-rw-------  1 root root 1675 Oct 28 13:09 id_rsa
-rw-------  1 root root  398 Oct 28 13:09 id_rsa.pub
-rw-r--r--  1 root root  413 Oct 28 11:49 known_hosts

When I try to connect to the git machine:

OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to git.mygitlab.com [212.29.122.24] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'git.mygitlab.com' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-    mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-    mic,password
debug1: Trying private key: /root/.ssh/identity
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Next authentication method: password
[email protected]'s password:

When I add SSH keys through the web interface, they aren't added to .ssh/authorized_keys.

I don't really know what to try next :(

Answer 1

If keys that you add to GitLab aren't making it into .ssh/authorized_keys:

1. Make sure that sidekiq is running. Keys are added to gitlab-shell in a Sidekiq worker, so if Sidekiq is down or backlogged, they won't make it in. You can check this in the output of ps -fu git and by checking the "background jobs" tab on the admin page.
2. Ensure that GitLab can execute gitlab-shell properly. The Sidekiq worker adds keys by exec'ing a gitlab-shell process. In particular, this won't work if the ssh_user setting is incorrect in gitlab.yml, or if gitlab-shell is installed to a location other than ~/gitlab-shell for that user.
3. Verify that the server's /home partition isn't full. If the disk that the authorized_keys file is stored on fills up, key appends with fail! This one has gotten me a few times. Use df -h /home to see if you still have room.

Check your logs for error messages from gitlab-shell: depending on the problem, error messages could appear in unicorn's or sidekiq's logs.