Continue training style-gan 2 network after crash

Question

I've been trying to train a style-gan2 network using a custom dataset. Unfortunately the server I'm currently running the computations on is somewhat unstable, causing it to crash after three days of training. Is there any way for me to continue training the network using the last snapshot of the network before it crashed? I have seen some references to continued training of a network, but neither the style-gan or style-gan2 github pages mention it.

Answer 1

After diggin through the code a bit I figured it out. Turns out there is a resume_pkl variable in training\training_loop. By setting that variable to the path of the snapshot I wanted to resume from I was able to restart the training process. The network has currently resumed training, I'll make another comment here if I encounter any further issues.