How to change how frequently SLURM updates the output file (stdout)?

Question

I am using SLURM to dispatch jobs on a supercomputer. I have set the --output=log.out option to place the content from a job's stdout into a file (log.out). I'm finding that the file is updated every 30-60 minutes, making it difficult for me to check on the status of my jobs.

Any idea why it takes so long to update this file? Is there a way to change settings so that this file is updated more frequently?

Using SLURM 14.03.4-2

Answer 1

This may be related to buffering.

Have you tried disabling output buffering as suggested in here? I would recommend the stdbuf option:

stdbuf -o0 -e0 command

But can't be sure without more information, as I've never experienced a behavior like that. Which filesystem are you using?

Also if you are using srun to run your commands you can use the --unbuffered option which disables the output buffering.