How to SSH into a GCE Instance created from a custom image?

Question

I'm having issues using ssh to log in to a VM created from a custom image.

I followed the steps for creating an image from an existing GCE instance.

I have successfully created the image, uploaded it to Google Cloud Storage and added it as an image to my project, yet when I try to connect to the new image, I get a "Connection Refused".

I can see other applications running on other ports for the new image, so it seems to be just ssh that is affected.

The steps I did are below:

...create an image from existing GCE instance (one I can log into fine via ssh)..then:

gcutil --project="river-ex-217" addimage example2 http://storage.googleapis.com/example-image/f41aca6887c339afb0.image.tar.gz
gcutil --project="river-ex-217" addinstance --image=example2 --machinetype=n1-standard-1 anothervm
gcutil --service_version="v1" --project="river-ex-217" ssh --zone="europe-west1-a" "anothervm"

Which outputs:

INFO: Running command line: ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /Users/mark1/.ssh/google_compute_engine -A -p 22 [email protected] --
ssh: connect to host 23.251.133.2 port 22: Connection refused

I've tried deleting the sshKeys metadata as suggested in another SO answer, and reconnecting which did this:

INFO: Updated project with new ssh key. It can take several minutes for the instance to pick up the key.
INFO: Waiting 120 seconds before attempting to connect.
INFO: Running command line: ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /Users/mark1/.ssh/google_compute_engine -A -p 22 [email protected] --
ssh: connect to host 23.251.133.2 port 22: Connection refused

I then try for the first instance in another zone, it works fine with the new key:

gcutil --service_version="v1" --project="river-ex-217" ssh --zone="europe-west1-b" "image1"

Both instances are running on the same "default" network with port 22 running, and ssh works for the first instance the image is created from.

I tried nc command from the other instance and my local machine, it shows no output:

nc 23.251.133.2 22

...whilst the original VM's ip shows this output:

nc 192.157.29.255 22
SSH-2.0-OpenSSH_6.0p1 Debian-4

I've tried remaking the image again and re-adding the instance, no difference.

I've tried logging in to the first instance, and switching user to one on that machine (which should be the same as the second machine?), and ssh from there.

WARNING: You don't have an ssh key for Google Compute Engine. Creating one now...
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
INFO: Updated project with new ssh key. It can take several minutes for the instance to pick up the key.
INFO: Waiting 300 seconds before attempting to connect.
INFO: Running command line: ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /home/mark/.ssh/google_compute_engine -A -p 22 [email protected] -- --zone=europe-west1-a
ssh: connect to host 23.251.133.2 port 22: Connection refused

I'm out of ideas, any help greatly appreciated :) The maddening thiing is I can see the new VM is live with the application ready, I just need to add a few files to it and set up some cronjobs. I guess I could do this pre-image making, but I would like to be able to log in at a later date and modify it, without needing to take 1hr to create images and launch new instances every time.

Yours faithfully, Mark

Answer 1

This question appears to be about how to debug SSH connectivity problems with images, so here is my answer to that.

It appears that your instance may not be running the SSH server properly. There may be something amiss with the prepared image.

Possibly useful debugging questions to ask yourself:

• Did you use gcimagebundle to bundle up the image or did it manually? Consider using the tool to make sure there isn't something you missed.
• Did you change anything about the ssh server configuration before bundling the image?
• When the instance is booting, check it's console output for ssh messages - it should mention regenerating the keys, starting sshd daemon and listening on port 22. If it does not or complains about something related to ssh, you should follow up on that.

You covered these, but for sake of completeness, these should also be checked:

• Can you otherwise reach the VM after it comes up? Does it respond on webserver ports (if any) or respond to ping?
• Double check that the network you VM is on allows SSH (port 22) access from the host you are connecting from.

You can compare your ssh setup to that of a working image:

• Create a new disk (disk-mine-1) from your image.
• Create a new disk (disk-upstream-1) from any working boot image, for example the debian wheezy one.
• Attach both of these to a VM you can access (either on console or from cli).
• SSH into the VM.
• Mount both of the images (sudo mkdir /mnt/{mine,upstream} && sudo mount /dev/sdb1 /mnt/mine && sudo mount /dev/sdc1 /mnt/upstream). Note that whether your image is sdb or sdc depends on the order you attached the images!
• Look for differences between the ssh config (diff -waur /mnt/{mine,upstream}/etc/ssh). There should not be any unless you specifically need them.
• Also check if your image has proper /mnt/mine/etc/init.d/{ssh,generate-ssh-hostkeys} scripts. They should also be linked from /mnt/mine/etc/rc{S,2}.d (S10generate-ssh-hostkeys and S02ssh respectively).