How to add timestamp to STDERR redirection

Question

If you're talking about an up-to-date timestamp on each line, that's something you'd probably want to do in your actual script (but see below for a nifty solution if you have no power to change it). If you just want a marker date on its own line before your script starts writing, I'd use:

( date 1>&2 ; myscript.sh ) 2>error.log

What you need is a trick to pipe stderr through another program that can add timestamps to each line. You could do this with a C program but there's a far more devious way using just bash.

First, create a script which will add the timestamp to each line (called predate.sh):

#!/bin/bash
while read line ; do
    echo "$(date): ${line}"
done

For example:

( echo a ; sleep 5 ; echo b ; sleep 2 ; echo c ) | ./predate.sh

produces:

Fri Oct  2 12:31:39 WAST 2009: a
Fri Oct  2 12:31:44 WAST 2009: b
Fri Oct  2 12:31:46 WAST 2009: c

Then you need another trick that can swap stdout and stderr, this little monstrosity here:

( myscript.sh 3>&1 1>&2- 2>&3- )

Then it's simple to combine the two tricks by timestamping stdout and redirecting it to your file:

( myscript.sh 3>&1 1>&2- 2>&3- ) | ./predate.sh >error.log

The following transcript shows this in action:

pax> cat predate.sh
    #!/bin/bash
    while read line ; do
        echo "$(date): ${line}"
    done
pax> cat tstdate.sh
    #!/bin/bash
    echo a to stderr then wait five seconds 1>&2
    sleep 5
    echo b to stderr then wait two seconds 1>&2
    sleep 2
    echo c to stderr 1>&2
    echo d to stdout
pax> ( ( ./tstdate.sh ) 3>&1 1>&2- 2>&3- ) | ./predate.sh >error.log
    d to stdout
pax> cat error.log
    Fri Oct  2 12:49:40 WAST 2009: a to stderr then wait five seconds
    Fri Oct  2 12:49:45 WAST 2009: b to stderr then wait two seconds
    Fri Oct  2 12:49:47 WAST 2009: c to stderr

As already mentioned, predate.sh will prefix each line with a timestamp and the tstdate.sh is simply a test program to write to stdout and stderr with specific time gaps.

When you run the command, you actually get "d to stdout" written to stderr (but that's your TTY device or whatever else stdout may have been when you started). The timestamped stderr lines are written to your desired file.

The devscripts package in Debian/Ubuntu contains a script called annotate-output which does that (for both stdout and stderr).

$ annotate-output make
21:41:21 I: Started make
21:41:21 O: gcc -Wall program.c
21:43:18 E: program.c: Couldn't compile, and took me ages to find out
21:43:19 E: collect2: ld returned 1 exit status
21:43:19 E: make: *** [all] Error 1
21:43:19 I: Finished with exitcode 2

Here's a version that uses a while read loop like pax's, but doesn't require extra file descriptors or a separate script (although you could use one). It uses process substitution:

myscript.sh 2> >( while read line; do echo "$(date): ${line}"; done > error.log )

Using pax's predate.sh:

myscript.sh 2> >( predate.sh > error.log )

The program ts from the moreutils package pipes standard input to standard output, and prefixes each line with a timestamp.

To prefix stdout lines: command | ts
To prefix both stdout and stderr: command 2>&1 | ts

I like those portable shell scripts but a little disturbed that they fork/exec date(1) for every line. Here is a quick perl one-liner to do the same more efficiently:

perl -p -MPOSIX -e 'BEGIN {$!=1} $_ = strftime("%T ", localtime) . $_'

To use it, just feed this command input through its stdin:

(echo hi; sleep 1; echo lo) | perl -p -MPOSIX -e 'BEGIN {$|=1} $_ = strftime("%T ", localtime) . $_'