Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Automatically "stop" Sagemaker notebook instance after inactivity?

I have a Sagemaker Jupyter notebook instance that I keep leaving online overnight by mistake, unnecessarily costing money...

Is there any way to automatically stop the Sagemaker notebook instance when there is no activity for say, 1 hour? Or would I have to make a custom script?

like image 971
p_mcp Avatar asked Dec 04 '18 09:12

p_mcp


2 Answers

You can use Lifecycle configurations to set up an automatic job that will stop your instance after inactivity.

There's a GitHub repository which has samples that you can use. In the repository, there's a auto-stop-idle script which will shutdown your instance once it's idle for more than 1 hour.

What you need to do is

  1. to create a Lifecycle configuration using the script and
  2. associate the configuration with the instance. You can do this when you edit or create a Notebook instance.

If you think 1 hour is too long you can tweak the script. This line has the value.

like image 118
Sanghyun Lee Avatar answered Oct 06 '22 15:10

Sanghyun Lee


You could also use CloudWatch + Lambda to monitor Sagemaker and stop when your utilization hits a minimum. Here is a list of what's available in CW for SM: https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html.

For example, you could set a CW alarm to trigger when CPU utilization falls below ~5% for 30 minutes and have that trigger a Lambda which would shut down the notebook.

like image 24
Matthew Arthur Avatar answered Oct 06 '22 16:10

Matthew Arthur