SGE script: print to file during execution (not just at the end)?

Question

I have an SGE script to execute some python code, submitted to the queue using qsub. In the python script, I have a few print statements (updating me on the progress of the program). When I run the python script from the command line, the print statements are sent to stdout. For the sge script, I use the -o option to redirect the output to a file. However, it seems that the script will only send these to the file after the python script has completed running. This is annoying because (a) I can no longer see real time updates on the program and (b) if my job does not terminate correctly (for example if my job gets kicked off the queue) none of the updates are printed. How can I make sure that the script is writing to the file each time it I want to print something, as opposed to lumping it all together at the end?

Answer 1

I think you are running into an issue with buffered output. Python uses a library to handle it's output, and the library knows that it's more efficient to write a block at a time when it's not talking to a tty.

There are a couple of ways to work around this. You can run python with the "-u" option (see the python man page for details), for example, with something like this as the first line of your script:

#! /usr/bin/python -u

but this doesn't work if you are using the "/usr/bin/env" trick because you don't know where python is installed.

Another way is to reopen the stdout with something like this:

import sys 
import os 

# reopen stdout file descriptor with write mode 
# and 0 as the buffer size (unbuffered) 
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) 

Note the bufsize parameter of os.fdopen being set to 0 to force it to be unbuffered. You can do something similar with sys.stderr.

Answer 2

As others mentioned, it is out of performance reasons to not always write the stdout when not connected to a tty.

If you have a specific point at which you want the stdout to be written, you can force that by using

import sys
sys.stdout.flush()

at that point.