How do I receive notification if a Google Compute Engine instance restarts or migrates on maintenance?

Question

I would love to receive an email (or other form of notification) every time my Compute Engine VM restarts, so I can connect and see that everything has started up again properly (and generate a general feel for the reliability of my instance)

I've looked through all the options in the Google Cloud Console, but haven't found anything...

Answer 1

If you have not already, go through the Live Migration docs to understand how Live migration works in general.

Live migrate

Google provides controls to set the instance availability policies which also lets you control aspects of live migration. Here they also mention what to look for to determine when live migration has taken place.

By default, standard instances are set to live migrate, where Google Compute Engine automatically migrates your instance away from an infrastructure maintenance event, and your instance remains running during the migration. Your instance might experience a short period of decreased performance, although generally most instances should not notice any difference. This is ideal for instances that require constant uptime, and can tolerate a short period of decreased performance.

When Google Compute Engine migrates your instance, it reports a system event that is published to the list of zone operations. You can review this event by performing a gcloud compute operations list --zones ZONE request or by viewing the list of operations in the Google Cloud Platform Console, or through an API request. The event will appear with the following text:

compute.instances.migrateOnHostMaintenance

Getting Live Migration Notices

In addition, you can detect directly on the VM when a maintenance event is about to happen.

The metadata server provides information about an instance's scheduling options and settings, through the scheduling/ directory and the maintenance-event attribute. You can use these attributes to learn about a virtual machine instance's scheduling options, and use this metadata to notify you when a maintenance event is about to happen through the maintenance-event attribute. By default, all virtual machine instances are set to live migrate so the metadata server will receive maintenance event notices before a VM instance is live migrated. If you opted to have your VM instance terminated during maintenance, then Compute Engine will automatically terminate and optionally restart your VM instance if the automaticRestart attribute is set. To learn more about maintenance events and instance behavior during the events, read about scheduling options and settings.

You can learn when a maintenance event will happen by querying the maintenance-event attribute periodically. The value of this attribute will change 60 seconds before a maintenance event starts, giving your application code a way to trigger any tasks you want to perform prior to a maintenance event, such as backing up data or updating logs. Compute Engine also offers a sample Python script to demonstrate how to check for maintenance event notices.

You can use the maintenance-event attribute with the waiting for updates feature to notify your scripts and applications when a maintenance event is about to start and end. This lets you automate any actions that you might want to run before or after the event. The following Python sample provides an example of how you might implement these two features together.

Terminate and (optionally) restart

If you want to disable live migration, you can choose to terminate and optionally also restart your instance when a maintenance event takes place.

If you do not want your instance to live migrate, you can choose to terminate and optionally restart your instance. With this option, Google Compute Engine will signal your instance to shut down, wait for a short period of time for your instance to shut down cleanly, terminate the instance, and restart it away from the maintenance event. This option is ideal for instances that demand constant, maximum performance, and your overall application is built to handle instance failures or reboots.

Look at the Setting availability policies section for more details on how to configure this.

Live migration and GPUs

If you use an instance with a GPU or a preemptible instance be aware that live migration is not supported:

Instances with GPUs attached cannot be live migrated. They must be set to terminate and optionally restart. Compute Engine offers a 60 minute notice before a VM instance with a GPU attached is terminated. To learn more about these maintenance event notices, read Getting live migration notices.

To learn more about handling host maintenance with GPUs, read Handling host maintenance on the GPUs documentation.

Live migration for preemptible instances

You cannot configure a preemptible instances to live migrate. The maintenance behavior for preemptible instances is always set to TERMINATE by default, and you cannot change this option. It is also not possible to set the automatic restart option for preemptible instances.

Note: For instances with GPUs, the attribute changes 60 minutes before the instances are terminated to give you time to shutdown and restart again on another host. Instances with GPUs are not live migrated and are instead terminated and optionally restarted. To learn more, read Handling host maintenance on the GPUs documentation.