bash: How to prepend a string to stderr lines and combine both stdout and stderr in exact order and store in one variable in bash?

Question

What I have done so far is :

#!/bin/bash

exec 2> >(sed 's/^/ERROR= /')

var=$(
        sleep 1 ; 
        hostname ; 
        ifconfig | wc -l ; 
        ls /sfsd; 
        ls hasdh;
        mkdir /tmp/asdasasd/asdasd/asdasd;
        ls /tmp ;
) 

echo "$var"

This does prepend ERROR= at the start of each error lines, but displays all errors first and then stdout, (not in order in which it was executed).

If we skip storing the output in variable and execute the commands directly, the output comes in desired order.

Any expert opinion would be appreciated.

Answer 1

The primary problem with your script is that the command substitution $(...) only captures the subshell's standard output; the subshell's standard error still just flows through to the parent shell's standard error. As it happens, you've redirected the parent shell's standard error in a way that ends up populating the parent shell's standard output; but that completely circumvents the $(...), which is only capturing the subshell's standard output.

Do you see what I mean?

So, you can fix that by redirecting the subshell's standard error in a way that ends up populating its standard output, which is what gets captured:

var=$(
    exec 2> >(sed 's/^/ERROR= /')
    sleep 1
    hostname
    ifconfig | wc -l
    ls /sfsd
    ls hasdh
    mkdir /tmp/asdasasd/asdasd/asdasd
    ls /tmp
)

echo "$var"

Even so, this does not guarantee proper ordering of lines. The problem is that sed is running in parallel with everything else in the subshell, so while it's just received an error-line and is busy planning to write to standard output, one of the later commands in the subshell can be plowing ahead and already writing more things to standard output!

You can improve that by launching sed separately for each command, so that the shell will wait for sed to complete before proceeding to the next command:

var=$(
    sleep 1 2> >(sed 's/^/ERROR= /')
    hostname 2> >(sed 's/^/ERROR= /')
    { ifconfig | wc -l ; } 2> >(sed 's/^/ERROR= /')
    ls /sfsd 2> >(sed 's/^/ERROR= /')
    ls hasdh 2> >(sed 's/^/ERROR= /')
    mkdir /tmp/asdasasd/asdasd/asdasd 2> >(sed 's/^/ERROR= /')
    ls /tmp 2> >(sed 's/^/ERROR= /')
)

echo "$var"

Even so, sed will be running concurrently with each command, so if any of those commands is a complicated command that writes both to standard output and to standard error, then the order that that command's output is captured in may not match the order in which the command actually wrote it. But this should probably be good enough for your purposes.

You can improve the readability a bit by creating a wrapper function for the simple-command (non-pipeline) case:

var=$(
    function fix-stderr () {
       "$@" 2> >(sed 's/^/ERROR= /')
    }

    fix-stderr sleep 1
    fix-stderr hostname
    fix-stderr eval 'ifconfig | wc -l'   # using eval to get a simple command
    fix-stderr ls /sfsd
    fix-stderr ls hasdh
    fix-stderr mkdir /tmp/asdasasd/asdasd/asdasd
    fix-stderr ls /tmp
)

echo "$var"