Google Cloud SSH Keys

Question

I have set up my new server with Google Compute Engine. I added a user and their public key into the metadata in the Google Cloud console (sshKeys).

I attempted to replace a public key in the metadata, but now the old one seems to be the only one able to ssh into my server (using PuTTY). The new one doesn't seem to be updated.

Now, even if I remove the whole metadata or type gibberish text into the sshKeys field, it will still work!

Could it be that it will require sometime for the metadata to be pushed to the server (my previous attempts were instantaneous)?

Answer 1

To understand how Google Compute Engine manages the ssh keys, you have to understand how GCE manages the metadata (since, as you wrote, they are in the metadata store).

And more specifically, the difference between project and instance metadata is crucial. To quote the documentation (see previous links):

Metadata can be assigned at both the project and instance level. Project level metadata propagates to all virtual machine instances within the project, while instance level metadata only impacts that instance. You can set both project and instance level metadata but if you set the same key for both your project and instance metadata, Compute Engine will use the instance metadata.

While this seems rather logical and straightforward, one has to pay attention, very closely, to the used terms:

Project level metadata propagates to all virtual machine instances within the project [...]

and

You can set both [...] but if you set the same key for both [...], Compute Engine will use the instance metadata.

If you consider both assertions, it means two things:

1. If you set the metadata at the project level ONLY, it will propagate in your instances.
2. If you set the metadata at the instance level, it will take precedence over the project level ones, and nothing will be propagated.

As a direct consequence of that, the GCE platform takes care of placing/removing your ssh keys in the instance (and creating the relevant users when placing them, while just removing the key from the ~user/.ssh/authorized_keys file when removing them - so you don't lose any data for ~user) ONLY when you do not specify your own keys (at instance creation or later). If you do, the GCE platform will consider the ssh key management as manual, and nothing will be kept in sync with the metadata store.

Fortunately, the GCE platform is well done, and therefore, you need not re-creating an instance to get your keys managed by the GCE platform: you only need to remove the instance level metadata relative to the sshKeys.

The same way, if you add some instance level metadata with the key sshKeys, it will disable the ssh keys GCE platform management; unless you remove that instance level metadata.

Concerning the delay question: I didn't have any delay other than the network delay (so no platform execution delay noticeable) so far. I don't think it's impossible that the platform has delays from time to time, but it doesn't seem very likely to be the cause of your problem.

---

Additional note:

Some distributions (such as ubuntu) include a specific user (in the case of ubuntu: ~ubuntu), with which every user existing in the project-level ssh keys can login; but that user's authorized_keys is generated at instance creation time, and never seems to be changed again by the GCE platform. IMHO, the automatic ssh key management should be preferred.

---

Source: personal experience with GCE, terraform, and the Google Developer Console

Answer 2

Removing a key from the sshKeys metadata does not cause the key to be removed from the instances.

It sounds like there was probably a mistake in the formatting of the new key which is why the new key was not added. Each line of the sshKeys metadata should look like "username:ssh-rsa AAAAB3Nza.....sjr comment". There should be not extra space, blank lines.

One common mistake is if you copy from the sshKeys metadata in the developer console, add a key, and paste it back you must be very careful with new lines, as the developer console loses the new lines in the html output.