Merge child process stdout and stderr

Question

How do I merge child process stdout and stderr?

The following does not work since ownership cannot be shared between stdout and stderr:

let pipe = Stdio::piped();
let prog = Command::new("prog")
                        .stdout(pipe)
                        .stderr(pipe)
                        .spawn()
                        .expect("failed to execute prog");

In other words, what is the Rust equivalent of 2>&1 in the shell?

Answer 1

My duct crate supports this:

#[macro_use]
extern crate duct;

fn main() {
    cmd!("echo", "hi").stderr_to_stdout().run();
}

The "right way" to do something like this, which duct is doing for you under the covers, is to create a double-ended OS pipe and pass the write end of it to both stdout and stderr. The standard library's Command class supports this sort of thing in general because Stdio implements FromRawFd, but unfortunately the standard library doesn't expose a way to create pipes. I've written another crate called os_pipe to do this inside of duct, and if you want you can use it directly.

This has been tested on Linux, Windows, and macOS.

Answer 2

I see nothing in the standard library that does this for you. Doesn't mean you can't write it yourself. This also means you get to decide how frequently each file descriptor is read and how to combine the data from each of the file descriptors. Here, I attempt to read in chunks using the default BufReader size and prefer to put stdout data first when both descriptors have data.

use std::io::prelude::*;
use std::io::BufReader;
use std::process::{Command, Stdio};

fn main() {
    let mut child =
        Command::new("/tmp/output")
        .stdout(Stdio::piped())
        .stderr(Stdio::piped())
        .spawn()
        .expect("Couldn't run program");

    let mut output = Vec::new();

    // Should be moved to a function that accepts something implementing `Write`
    {
        let stdout = child.stdout.as_mut().expect("Wasn't stdout");
        let stderr = child.stderr.as_mut().expect("Wasn't stderr");

        let mut stdout = BufReader::new(stdout);
        let mut stderr = BufReader::new(stderr);

        loop {
            let (stdout_bytes, stderr_bytes) = match (stdout.fill_buf(), stderr.fill_buf()) {
                (Ok(stdout), Ok(stderr)) => {
                    output.write_all(stdout).expect("Couldn't write");
                    output.write_all(stderr).expect("Couldn't write");

                    (stdout.len(), stderr.len())
                }
                other => panic!("Some better error handling here... {:?}", other)
            };

            if stdout_bytes == 0 && stderr_bytes == 0 {
                // Seems less-than-ideal; should be some way of
                // telling if the child has actually exited vs just
                // not outputting anything.
                break;
            }

            stdout.consume(stdout_bytes);
            stderr.consume(stderr_bytes);
        }
    }

    let status = child.wait().expect("Waiting for child failed");
    println!("Finished with status {:?}", status);
    println!("Combined output: {:?}", std::str::from_utf8(&output))
}

The biggest gap is telling when the process has exited. I'm surprised by the lack of a relevant method on Child.

See also How do I prefix Command stdout with [stdout] and [sterr]?

---

In this solution, there isn't any intrinsic ordering between the file descriptors. As an analogy, imagine two buckets of water. If you empty a bucket and later see that it's been filled up again, you know the second bucket came after the first. However, if you empty two buckets and come back later and both are filled up, you can't tell which bucket was filled first.

The "quality" of interleaving is a matter of how frequently you read from each file descriptor and which file descriptor is read first. If you read a single byte from each in a very tight loop, you might get completely garbled results but these would be the most "accurate" with regard to ordering. Likewise, if a program prints "A" to stderr then "B" to stdout but the shell reads from stdout before stderr, then the result would be "BA", which looks backwards.

Answer 3

I wrote the io-mux crate to provide a multi-ended pipe-like construct; the primary use case is to capture stdout and stderr from a process, properly interleaved, and distinguish which data came from which one. See highlight-stderr for an example of how to use it.

(io-mux primarily works on Linux; it can work on other UNIX platforms as well, but there it has some limitations that arise from the behavior of UNIX sockets on those platforms.)

If you don't care about distinguishing which data came from stdout and which data came from stderr, you can use a normal pipe. On UNIX, use libc::pipe to create a pipe, Stdio::from_raw_fd twice to create stdout and stderr, spawn the process, and then read from the other end of the pipe. On Windows, you can do something similar, using handles rather than file descriptors.

If you don't need to distinguish which data came from stdout and which data came from stderr, and you don't want to deal with the platform-specific details of setting up a pipe, try something like subprocess, which specifically mentions support for combining stdout and stderr.