How to save plot of live data on a remote machine?

Question

I want to know how my model is doing while it's training (i.e live data manner) by sshing and checking the plots.

Using the animation.FuncAnimation I'm able to make it save (& overwrite) a frame every time it updates on my local machine like this:

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation

def animate(i):
    fig.clf()

    plt.suptitle('Title here')

    # actually parse model outputs and plot
    values = np.random.random_integers(0, 100, (100,))
    plt.plot(np.arange(len(values)), values, label='random')
    plt.ylabel('demo data')
    plt.grid()


    plt.legend()
    plt.xlabel('Epochs')
    fig.savefig('Figure.png')


fig = plt.figure()
ani = animation.FuncAnimation(fig, animate, interval=10*60*1000)
plt.show()

Using this on a local machine is fine since the plt.show calls the $DISPLAY. However, when running on a remote server (via ssh of course) since there's no display, I get the RuntimeError: Invalid DISPLAY variable. When using other backends something like svg via matplotlib.use('svg'). The script exits without actually saving any image.

Also, my decision to use plt.show() and fig.savefig('Figure.png') inside the animate function were because without the plt.show() function after the call to FuncAnimation, it does NOT run the animate after given interval. I tried doing plt.savefig instead.

And regarding the fig.savefig('Figure.png'), doing so outside the animate function leads to a blank image. I'm guessing since I clear the image at the start of the animate function.

So, my question is: Is there a way to save figures generated on live data using animation (or FuncAnimation) like this over ssh till some event occurs (or perhaps a timeout)?

Answer 1

As an alternative to the webserver suggestion, you can use watch to run scp every n seconds and copy the png to your local machine.

Something like:

watch -n 5 scp user@remote:/path/to.png ./

which would copy the file every 5 seconds.

Or use rsync to only copy when it has actually changed. (There are also utilities that will use scp or rsync to copy it to your local machine when the file changes.)