Google cloud share data between VM's

Question

I wish to create a Virtual Machine on Google Compute Engine with GPU's in order to carry out certain tasks.

Now the problem is that the data to be uploaded to the VM's from my local network is huge, and I am severely restricted by my slow and unreliable internet. I am aware of file transfer between unix machines using SCP, but even that would take hours to take place, and that means my GPU's online (the most expensive components) would be sitting idle, and I would be charged for nothing.

I have thought of transferring files to the VM first and then adding GPU's, but I have been unable to edit the VM to add GPU's in such a case.

Hence, I need help with two possible solutions.

Is it possible for me to add a persistent disk to one VM, transfer data, spawn a new VM and shift the disk to the new VM? If yes then how?

OR

Is it possible to edit an existing VM to add GPU instances post it's creation? If yes then how?

Any sort of help will be appreciated. Thank You!

Answer 1

These are some suggestions:

• Upload your data to a Google Cloud Storage bucket and then pull data from there to the VM when needed (this will be relatively fast compared to copying data from your local machine into the VM). GCS also provides a FUSE tool to mount the GCS buckets on your VM and then be able to read/write data as needed. GCS buckets is the most flexible option in this list.

• You can create a persistent disk, upload the data into it once. Then attach it as needed to your GPU based VMs when needed. You can have multiple VMs attaching the same disk simultaneously in read-only mode.

• You can snapshot persistent disks, and restore a snapshot on to a new persistent disk if needed. This is more useful for backups than your primary use case.

• Consider chunking and/or sharding your data so that you can pipeline pulling data from a remote server (like GCS) when you're running your workloads on GPU in parallel.