Cannot SSH Google Cloud Instance -

Question

I've tried every method available in googles documentation - yet I still can't ssh into my compute engine instance on google cloud. Posting a log for context.

username@instancename:~$ gcloud compute ssh instancename --ssh-flag="-vvv"
For the following instances:
 - [instancename]
choose a zone:
 [1] asia-east1-c
 [2] asia-east1-a
 [3] asia-east1-b
 [4] asia-northeast1-b
 [5] asia-northeast1-c
 [6] asia-northeast1-a
 [7] europe-west1-c
 [8] europe-west1-b
 [9] europe-west1-d
 [10] us-central1-f
 [11] us-central1-a
 [12] us-central1-c
 [13] us-central1-b
 [14] us-east1-b
 [15] us-east1-d
 [16] us-east1-c
 [17] us-west1-b
 [18] us-west1-a
Please enter your numeric choice:  13 

OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 104.xxx.xxx.xx [104.xxx.xxx.xx] port 22.
debug1: Connection established.
debug1: identity file /home/username/.ssh/google_compute_engine type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/google_compute_engine-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Debian-4~bpo70+1
debug1: match: OpenSSH_6.6.1p1 Debian-4~bpo70+1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: using hostkeyalias: compute.14068955514934919297
debug3: load_hostkeys: loading entries for host "compute.14068955514934919297" from file "/home/username/.ssh/google_compute_known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/username/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha
1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijnd
[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijnd
[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha1-96-etm@ope
nssh.com,[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha1-96-etm@ope
nssh.com,[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: setup [email protected]
debug1: kex: server->client aes128-ctr [email protected] none
debug2: mac_setup: setup [email protected]
debug1: kex: client->server aes128-ctr [email protected] none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 23:66:fa:ae:3e:da:ec:f8:d3:ea:c8:c0:84:de:91:82
debug1: using hostkeyalias: compute.14068955514934919297
debug3: load_hostkeys: loading entries for host "compute.14068955514934919297" from file "/home/username/.ssh/google_compute_known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/username/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'compute.14068955514934919297' is known and matches the ECDSA host key.
debug1: Found key in /home/username/.ssh/google_compute_known_hosts:1
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/username/.ssh/google_compute_engine (0x7fc8787042f0), explicit
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/username/.ssh/google_compute_engine
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
username@instancename:~$

This is from a ssh attempt in the gcloud shell, I've also tried the cli tool and connecting through the VM. I've allowed gcloud to auto-generate keys and checked that both the private and public keyfiles exist. The instance is running and port 22 is open. I'm all out of ideas.

Answer 1

Below steps, will give you a serial access to your Google Cloud instance, from there you can Validate the Guest environment

I suggest you first to verify that SSH access to the instance is not blocked by a firewall.

gcloud compute firewall-rules list | grep "tcp:22"

Make sure you still have enough space disk left in the root volume,by running the following gcloud command in your shell.

gcloud compute instances get-serial-port-output [INSTANCE-NAME]

Look for some entry like

...No space left on device...

...google-accounts: ERROR Exception calling the response handler. [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']...

Connect to the instance using the serial console

1. Go to the VM instances page in Google Cloud Platform console. 
2. Click on the instance for which you want to add a startup script. 
3. Click the Edit button at the top of the page.
4. Click on ‘Enable connecting to serial ports’
5. Under Custom metadata, click Add item. 
6. Set 'Key' to 'startup-script' and set 'Value' to this script:

#! /bin/bash 
useradd -G sudo USERNAME 
echo 'USERNAME:PASSWORD' | chpasswd

7. Click Save and then click RESET on the top of the page. You might need to wait for some time for the instance to reboot. 
8. Click on 'Connect to serial port' in the page. 
9.  In the new window, you might need to wait a bit and press on Enter of your keyboard once; then, you should see the login prompt. 
10. Login using the USERNAME and PASSWORD you provided.

To validate the guest environment

Then inside the instance you need to fetch which is not working by Validate the Guest Environment :

First: look in your serial console if these line below are listed :

Started Google Compute Engine Accounts Daemon 
Started Google Compute Engine IP Forwarding Daemon 
Started Google Compute Engine Clock Skew Daemon 
Started Google Compute Engine Instance Setup 
Started Google Compute Engine Startup Scripts 
Started Google Compute Engine Shutdown Scripts 
Started Google Compute Engine Network Setup

Second: Verify if the package for the guest Environment is installed run the command in your serial output

apt list --installed | grep google-compute

It should list the below line :

google-compute-engine
google-compute-engine-oslogin
python-google-compute-engine
python3-google-compute-engine

Third:you need to verify if all the services for the guest environment are running by running this command :

sudo systemctl list-unit-files | grep google | grep enabled

It should list the below line :

google-accounts-daemon.service      enabled
google-ip-forwarding-daemon.service enabled
google-clock-skew-daemon.service    enabled
google-instance-setup.service       enabled
google-shutdown-scripts.service     enabled
google-startup-scripts.service      enabled
google-network-setup.service        enabled