Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

GitLab SSH keys stopped working

OK, I've stumbled on this topic many times, but this is the first time that none of the regular solutions worked.

I have a CentOS 6.4 server running GitLab. It's been working great with 20+ users and 60+ projects, but about 5 hours ago, my main staging server was unable to connect to the GitLab machine for the first time using key authentication and it prompted for password. I regenerated the RSA key and added it to my deploy keys, but that failed as well.

Next, I tried to create a new user on the staging server, create a key for it, and add it to GitLab but it still fails.

Permissions:

drwxr-x---  22 root root 4.0K Oct 28 13:20 root

Inside root:

drwx------   2 root root     4096 Oct 28 11:49 .ssh

Inside .ssh:

-rw-------  1 root root  227 Oct 28 11:48 authorized_keys
-rw-------  1 root root 1675 Oct 28 13:09 id_rsa
-rw-------  1 root root  398 Oct 28 13:09 id_rsa.pub
-rw-r--r--  1 root root  413 Oct 28 11:49 known_hosts

When I try to connect to the git machine:

OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to git.mygitlab.com [212.29.122.24] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'git.mygitlab.com' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-    mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-    mic,password
debug1: Trying private key: /root/.ssh/identity
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Next authentication method: password
[email protected]'s password:

When I add SSH keys through the web interface, they aren't added to .ssh/authorized_keys.

I don't really know what to try next :(

like image 639
SlipY Avatar asked Oct 28 '13 12:10

SlipY


People also ask

Why is my SSH key not working?

Make sure the authorized_keys file and the private key itself have the correct permissions and ownership. Check that key-based authentication is allowed by the server. Make sure the private key is readable by the SSH client. If you're using PuTTY, make sure your SSH keys are properly configured for the session.

Where does GitLab store SSH keys?

premium. The authorized_keys file contains all the public SSH keys for users allowed to access GitLab.

How do I recover my SSH key?

If you lose your SSH key passphrase, there's no way to recover it. You'll need to generate a brand new SSH keypair or switch to HTTPS cloning so you can use your GitHub password instead. If you lose your SSH key passphrase, there's no way to recover it.

Can I have 2 SSH keys GitLab?

The problem with multiple GitLab accounts is that you can't use one SSH key on both of them. For example, you have a personal email address associated with one GitLab account, and business email with another.


1 Answers

If keys that you add to GitLab aren't making it into .ssh/authorized_keys:

  1. Make sure that sidekiq is running. Keys are added to gitlab-shell in a Sidekiq worker, so if Sidekiq is down or backlogged, they won't make it in. You can check this in the output of ps -fu git and by checking the "background jobs" tab on the admin page.
  2. Ensure that GitLab can execute gitlab-shell properly. The Sidekiq worker adds keys by exec'ing a gitlab-shell process. In particular, this won't work if the ssh_user setting is incorrect in gitlab.yml, or if gitlab-shell is installed to a location other than ~/gitlab-shell for that user.
  3. Verify that the server's /home partition isn't full. If the disk that the authorized_keys file is stored on fills up, key appends with fail! This one has gotten me a few times. Use df -h /home to see if you still have room.

Check your logs for error messages from gitlab-shell: depending on the problem, error messages could appear in unicorn's or sidekiq's logs.

like image 147
Ash Wilson Avatar answered Sep 20 '22 15:09

Ash Wilson