Go: receive os.cmd stdout and stderr in order

Question

I need to execute subcommand from go and process it stdout and stderr separately, with keeping order of ouput that comes to stdin/stdout. I've tried several differents ways, but could not achieve the correct order of output; following code shows that ouput handling order is absolutely random:

package main

import (
    "fmt"
    "log"
    "os/exec"
)

var (
    result = ""
)

type writer struct {
    result string
    write  func(bytes []byte)
}

func (writer *writer) Write(bytes []byte) (int, error) {
    writer.result += string(bytes) // process result later
    result += string(bytes)
    return len(bytes), nil
}

func main() {
    cmd := exec.Command("bash", "-c", "echo TEST1; echo TEST2 1>&2; echo TEST3")

    stderr := &writer{}
    cmd.Stderr = stderr

    stdout := &writer{}
    cmd.Stdout = stdout

    err := cmd.Start()
    if err != nil {
        log.Fatal(err)
    }

    err = cmd.Wait()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(result)
}

With several runs code can output following:

$ go run main.go
TEST1
TEST3
TEST2

I expect following result in all cases:

$ go run main.go
TEST1
TEST2
TEST3

I can not call cmd.CombinedOutput because I need to process stdout/stderr separately and in realtime.

Answer 1

There is no "order" with the command that you're executing. They're parallel pipes, and as such they're effectively concurrent in the same way that two goroutines are concurrent. You can certainly store them in the order you receive them and tag them with their source by making use of channels or a mutex. In order to make the output not random with your synthetic example, you need to add a bit of a pause. I have used this method successfully with real commands, however:

package main

import (
    "fmt"
    "log"
    "os/exec"
    "sync"
)

var (
    result = ""
)

type write struct {
    source string
    data   string
}

type writer struct {
    source string

    mu     *sync.Mutex
    writes *[]write
}

func (w *writer) Write(bytes []byte) (int, error) {
    w.mu.Lock()
    defer w.mu.Unlock()
    *w.writes = append(*w.writes, write{
            source: w.source,
            data:   string(bytes),
    })
    return len(bytes), nil
}

func main() {
    cmd := exec.Command("bash", "-c", "echo TEST1; sleep .1; echo TEST2 1>&2; sleep .1; echo TEST3")

    var mu sync.Mutex
    var writes []write

    cmd.Stderr = &writer{
            source: "STDERR",
            mu:     &mu,
            writes: &writes,
    }
    cmd.Stdout = &writer{
            source: "STDOUT",
            mu:     &mu,
            writes: &writes,
    }

    err := cmd.Start()
    if err != nil {
            log.Fatal(err)
    }

    err = cmd.Wait()
    if err != nil {
            log.Fatal(err)
    }

    fmt.Printf("%q\n", writes)
}

will produce

[{"STDOUT" "TEST1\n"} {"STDERR" "TEST2\n"} {"STDOUT" "TEST3\n"}]