Google Cloud Engine. Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

Question

I can not connect via ssh any more, I was be able to connect for almost 24 hours. All of a sudden, ssh stops working. I had many users and I also added a new (tomcat) user in that VM.

I am get the below message when I try to ssh to my instance:

"Permission denied (publickey,gssapi-keyex,gssapi-with-mic)."

I ended up removing ~/.ssh/google_compute_engine*

Removed the 'sshKeys' metadata from Cloud Engine Console

Tried gcutil ssh again, this created new ~/.ssh/google_compute_engine file as well as sshKeys metadata.

But still I am getting that error.

Answer 1

I had the same problem and I was debugging it for about 16 hours. Yet that I have found the solution I'd like you to have a share in my odyssey.

I was running GitLab on Google Compute Engine advertised as one click installation.

Well, finally as I tried to clone a private repository, I got the error message:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I looked after the private/public key pair and I found nothing unusual.

---

Then I thought there might something wrong with the sshd on the server as I got the debug message:

debug1: ssh_rsa_verify: signature correct
[...]
debug1: Roaming not allowed by server

So I checked a plethora of different sshd settings but nothing really fixed the issue.

---

And at last I started debugging on the server side and found the error:

sshd[7364]: debug1: Could not open authorized keys '/var/opt/gitlab/.ssh/authorized_keys': Permission denied

Finally this was the highway to happiness. Because the file existed and sshd knew which file it has to load. However, somehow there was a permission issue.

So I checked if the chmod of the files in the remote .ssh folder were ok. I found nothing unusual.

---

And here is the solution:

SELinux did have a problem with the location of the .ssh folder and was not willing to give permission to the ssh daemon. By either executing the command

restorecon -Rv /var/opt/gitlab/.ssh/

or

semanage fcontext -a -t ssh_home_t "/var/opt/gitlab/.ssh/authorized_keys"

One of both commands solved the issue. I'll be glad if someone could verify which of those both!

So you don't need to deactivate SELinux!

Answer 2

This is really a comment on the correct solution by @sxleixer, but I wanted formatting.

1. The semanage tool is not installed by default. Go get it with

   sudo yum -y install policycoreutils-python
   

2. Allow the nonstandard ssh_home_t

   sudo semanage fcontext -a -t ssh_home_t "/var/opt/gitlab/.ssh/authorized_keys"
   

3. Either restart sshd or do a full restart with

   sudo shutdown -r now
   

4. Test everything is working locally

   ssh-keygen -t rsa -C "[email protected]"
   cat ~/.ssh/id_rsa.pub # Copy-paste the key to the 'My SSH Keys' section under the 'SSH' tab in your user profile
   ssh -T git@localhost  # Should now output "Welcome to GitLab"
   

This fixes the one click installation of GitLab on Google Compute Engine.

There is indeed no good reason to turn off SELinux.