Running multiple Colab notebooks in parallel over Google Cloud?

Question

I need to do a ton of processing in colab, and It'll take a really long time for 1-3 notebooks to do it all.

Is it possible to pay for more colab instances so I can run like 10-20 notebooks at a time? I think I saw something of this being possible on Google Cloud but I am hazy on the details. If this is possible, is there a guide which shows me how to do this?

Answer 1

See the section 'Connecting to a runtime on a Google Compute Engine instance' on this page:

https://research.google.com/colaboratory/local-runtimes.html

Reproducing here:

If the Jupyter notebook server you'd like to connect to is running on another machine (e.g. Google Compute Engine instance), you can set up SSH local port forwarding to allow Colaboratory to connect to it.

Note: Google Cloud Platform provides Deep Learning VM images with Colaboratory local backend support preconfigured. Follow the how-to guides to set up your Google Compute Engine instance with local SSH port forwarding. If you use these images, skip directly to Step 4: Connect to the local runtime (using port 8888).

First, set up your Jupyter notebook server using the instructions above.

Second, establish an SSH connection from your local machine to the remote instance (e.g. Google Compute Engine instance) and specify the '-L' flag. For example, to forward port 8888 on your local machine to port 8888 on your Google Compute Engine instance, run the following:

gcloud compute ssh --zone YOUR_ZONE YOUR_INSTANCE_NAME -- -L 8888:localhost:8888

Finally, make the connection within Colaboratory by connecting to the forwarded port (follow the same instructions under Step 4: Connect to the local runtime).