Reading from a processes stdout without placing it all in memory at once

Question

I am trying to capture the output from an external tool which is run in a separate process. I would like to do so in a non-blocking way as the output is larger than memory. I saw How would you stream output from a Process in Rust? but I am not sure how to proceed. I have copied an example from here but this seems to capture output into a variable before proceeding. So far I have:

let path = "/Documents/Ruststuff/DB30_igh_badPE.bam";
let output = Command::new("samtools")
    .arg("view")
    .arg("-H")
    .arg(path)
    .stdin(Stdio::piped())
    .stdout(Stdio::piped())
    .spawn()
    .unwrap_or_else(|e| panic!("failed {}", e));

let mut s = String::new();
match output.stdout.unwrap().read_to_string(&mut s) {
    Err(why) => panic!("{}", Error::description(&why)),
    Ok(_) => print!("{}", s),
}

Is it possible to loop over stdout from the child process instead of using the match? Something like:

for line in &output.stdout {}

Answer 1

You don't need non-blocking IO for what you want. You can use a buffered reader to loop over the lines of input. This assumes that you always need a full line, and that a full line isn't too much data:

use std::{
    io::{BufRead, BufReader},
    process::{Command, Stdio},
};

fn main() {
    let mut child = Command::new("yes")
        .stdout(Stdio::piped())
        .spawn()
        .expect("Unable to spawn program");

    if let Some(stdout) = &mut child.stdout {
        let lines = BufReader::new(stdout).lines().enumerate().take(10);
        for (counter, line) in lines {
            println!("{}, {:?}", counter, line);
        }
    }
}

ChildStdout implements Read for itself, but not for an immutable reference (&ChildStdout). Although we own the standard out, we don't want to consume it, so we need a reference of some kind. Read is implemented for a mutable reference to any other type that is itself Read, so we switch to a mutable reference. Then child needs to be mutable as well.