How to let SBATCH send stdout via email?

Question

I would like to let the slurm system send myprogram output via email when the computing is done. So I wrote the SBATCH as following

#!/bin/bash -l
#SBATCH -J MyModel
#SBATCH -n 1 # Number of cores
#SBATCH -t 1-00:00 # Runtime in D-HH:MM
#SBATCH -o JOB%j.out # File to which STDOUT will be written
#SBATCH -e JOB%j.err # File to which STDERR will be written
#SBATCH --mail-type=END
#SBATCH [email protected]
echo $SLURM_JOB_ID 
echo $SLURM_JOB_NAME 
/usr/bin/mpirun -np 1 ./myprogram
/usr/bin/mail -s $SLURM_JOB_NAME [email protected] < JOB${SLURM_JOB_ID}.out

The mail system reports

file .out not found

How can I construct the mail command to let the subject line be $SLURM_JOB_NAME and the mail contents from STDOUT file, e.g. JOB${SLURM_JOBID}.out in my case?

Answer 1

Here is my solution:

#!/bin/bash
#SBATCH -J MyModel
#SBATCH -n 1 # Number of cores
#SBATCH -t 1-00:00 # Runtime in D-HH:MM
#SBATCH -o JOB%j.out # File to which STDOUT will be written
#SBATCH -e JOB%j.out # File to which STDERR will be written
#SBATCH --mail-type=BEGIN
#SBATCH [email protected]

echo "$(date "+%Y-%m-%d %H:%M:%S"): $SLURM_JOB_NAME start id=$SLURM_JOB_ID"

/usr/bin/mpirun -np 1 ./myprogram

cat JOB${SLURM_JOB_ID}.out | mail -s "$SLURM_JOB_NAME Ended id=$SLURM_JOB_ID"" [email protected]

and further we can add more info and keep exit code:

#!/bin/bash
#SBATCH -J MyModel
#SBATCH -n 1 # Number of cores
#SBATCH -t 1-00:00 # Runtime in D-HH:MM
#SBATCH -o JOB%j.out # File to which STDOUT will be written
#SBATCH -e JOB%j.out # File to which STDERR will be written
#SBATCH --mail-type=BEGIN
#SBATCH [email protected]

secs_to_human(){
    echo "$(( ${1} / 3600 )):$(( (${1} / 60) % 60 )):$(( ${1} % 60 ))"
}
start=$(date +%s)
echo "$(date -d @${start} "+%Y-%m-%d %H:%M:%S"): ${SLURM_JOB_NAME} start id=${SLURM_JOB_ID}\n"

### exec task here
( << replace with your task here >> ) \
&& (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Ended after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" [email protected] && echo mail sended) \
|| (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Failed after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" [email protected] && echo mail sended && exit $?)

you can also edit this to send seperate stdout/stderr logs or attach them as files.

This snippet is shared on github-gists