Running a python script on Google Cloud Compute Engine

Question

For a machine learning task at school I wrote my own MLP network. The data set is quite big, and training takes forever. I was alerted to the option of running my script on the Google Cloud Compute Engine. I tried to set this up, but did not succeed (yet).

The steps I undertook where:

1. Create an account
2. Create a VM
3. Open the VM via the browser

Can anyone help me with importing and running my python script into the Google Cloud. Or does anyone have clear a tutorial on how to solve this? I tried finding these myself, but had no success so far.

Answer 1

I finally figured this out so I'll post the same answer on my own post that worked for me here. Using Debian Stretch on my VM. I'm assuming you already uploaded your file(s) to the VM and that you are in the same directory of your script.

1. Make your script an executable

   chmod +x myscript.py
   

2. Run the nohup command to execute the script in the background. The & option ensures that the process stays alive after exiting. I've added the shebang line to my python script so there's no need to call python here

   nohup /path/to/script/myscript.py &
   

3. Logout from the shell if you want

   logout
   

Done! Now your script is up and running. You can login back and make sure that your process is still alive by checking the output of this command:

ps -e | grep myscript.py

If anything went wrong, you can check out the nohup.out file to see the output of your script:

cat nohup.out

Answer 2

There is even a simpler approach to to run code in the background in gcp and in every linux terminal: using screen linux

Create a new background terminal window:

screen -S WRITE_A_NAME_OF_YOUR_CHOIC_HERE

now you are in a background window in the terminal. Run your code:

python3 mycode.py

Exit screen with the hotkeys and the job will keep running on the background.

ctrl + A + D

You can close all windows now. If you wanna go back and see what's happening. Log again in your terminal. And tap the following.

screen -ls 

This one will give you the list of the created "windows". Now find yours and tap

screen -r WRITE_NAME_OF_YOUR_WINDOW

And there you have it :D You can find more commands here