SSH Key asks for password

Question

I stuck here now for like 2 Days a week.

I've got a CentOs machine with Gitlab4 and gitolite. Everything worked fine for weeks, but suddenly last weekend something strange happend quite all binaries disappeared from the mashine ( like yum, python, ruby, mysql ect. ) i've really no clue how that can happn... After hours of reinstalling and compiling gitlab was working again.

But i cant get the ssh keys between the gitlab and git user working. I already deleted and recreated the git user, set again all permissions, recreated the ssh keys, reinstalld gitolite ect. But nothing worked i keep getting the same error.

git user .ssh folder

-rwx------ 1 git git  557 Mar 27 16:46 authorized_keys

gitlab user .ssh folder

-rw------- 1 gitlab gitlab 1671 Mar 27 16:45 id_rsa
-rw-r--r-- 1 gitlab gitlab  406 Mar 27 16:45 id_rsa.pub
-rw-r--r-- 1 gitlab gitlab  391 Mar 27 16:50 known_hosts

SSH error:

ssh -vvvT git@localhost
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/gitlab/.ssh/identity type -1
debug3: Not a RSA1 key file /home/gitlab/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/gitlab/.ssh/id_rsa type 1
debug1: identity file /home/gitlab/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3p2
debug1: match: OpenSSH_4.3p2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3p2
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 132/256
debug2: bits set: 502/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /home/gitlab/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host 'localhost' is known and matches the RSA host key.
debug1: Found key in /home/gitlab/.ssh/known_hosts:1
debug2: bits set: 505/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/gitlab/.ssh/identity ((nil))
debug2: key: /home/gitlab/.ssh/id_rsa (0x848ba50)
debug2: key: /home/gitlab/.ssh/id_dsa ((nil))
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/gitlab/.ssh/identity
debug3: no such identity: /home/gitlab/.ssh/identity
debug1: Offering public key: /home/gitlab/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/gitlab/.ssh/id_dsa
debug3: no such identity: /home/gitlab/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

The auth log gives me:

Apr  2 10:19:13 venus sshd[15693]: User git not allowed because account is locked
Apr  2 10:19:13 venus sshd[15693]: Failed none for illegal user git from ::ffff:127.0.0.1 port 56906 ssh2

Thanks for any Help.

Answer 1

You mention:

Apr 2 10:19:13 venus shd[15693]: User git not allowed because account is locked 
Apr 2 10:19:13 venus sshd[15693]: Failed none for illegal user git from ::ffff:127.0.0.1 port 56906 ssh2

This article mentions:

OpenSSH now checks for locked accounts by default.
On Linux systems, locked accounts are defined as those that have !! in the password field of /etc/shadow.
This is the default entry for accounts created with the useradd command.
Even if you are using GSI authentication and do not need local passwords, sshd won't let the user login with this message:

Too many authentication failures for username

In the sshd debugging info it will indicate that the account is locked:

User username not allowed because account is locked

Here is some additional information from the sshd Manual:

Regardless of the authentication type, the account is checked to ensure that it is accessible.
An account is not accessible if it is locked, listed in DenyUsers or its group is listed in DenyGroups.
The definition of a locked account is system dependant.
Some platforms have their own account database (eg AIX) and some modify the passwd field ( "*LK*" on Solaris and UnixWare, "*" on HP-UX, containing "Nologin" on Tru64, a leading "*LOCKED*" on FreeBSD and a leading "!!" on Linux).
If there is a requirement to disable password authentication for the account while allowing still public-key, then the passwd field should be set to something other than these values (eg "NP" or "*NP*" ).

Fix: Replace !! with (for example) NP in /etc/shadow.

---

As mentioned by jszakmeister (comments) and Yongcan-Frank-Lv (comments):

sudo passwd -u git

would be enough to unlock the account.

Answer 2

This exact same issue was killing me in gitlab 5.2 (bitnami).

I finally tracked it down in /var/log/auth.log which showed:

May 28 11:32:10 ml115 sshd[27779]: User git not allowed because account is locked
May 28 11:32:10 ml115 sshd[27779]: input_userauth_request: invalid user git [preauth]

After that, it didn't take me long to find that the git entry in /etc/shadow had a ! that needed to be replaced with a *.

With * and all my keys set up, I was able to ssh in from another machine (note that ssh -vvT git@gitserver also helps with diagnosis).

git push -u origin master

now works.

My system is Ubuntu 13.04.