Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Duplicate stdout, pipe it to two different commands, collect results from both to stdin of final program

Tags:

shell

unix

pipe

Say I have three programs: generator, that produces input data fed to processor and verifier that can check if processor output is correct for given input (so it needs both files).

What I currently do is:

generator > in.txt && processor < in.txt > out.txt && cat in.txt out.txt | verifier

Is it possible to achieve the same result without using explicit files? I've read about duplicating input using tee and and process substitution, but I didn't find a way to collect both streams into single one for final step.

like image 358
Kim Strauss Avatar asked Feb 05 '13 10:02

Kim Strauss


People also ask

How can you use pipe in multiple commands?

You can make it do so by using the pipe character '|'. Pipe is used to combine two or more commands, and in this, the output of one command acts as input to another command, and this command's output may act as input to the next command and so on.

How does stdin and stdout work?

stdin − It stands for standard input, and is used for taking text as an input. stdout − It stands for standard output, and is used to text output of any command you type in the terminal, and then that output is stored in the stdout stream.

Which command is used within a pipe to send output to a file and across a pipe?

Pipe shell command It is used to pipe, or transfer, the standard output from the command on its left into the standard input of the command on its right. # First, echo "Hello World" will send Hello World to the standard output. # Next, pipe | will transfer the standard output to the next command's standard input.


2 Answers

I have not tested this, but try:

{ generator | tee /dev/stderr | processor ; } 2>&1  | verifier

This will redirect a copy of generator output to stderr. Then run processor on stdout of generator. Then combine both & pipe to verifier.

However, this cannot guarantee the order, in which lines from generator & processor would reach verifier.


Alternately, you can try process substitution like below:

( generator | tee >(processor) ) | verifier
like image 125
anishsane Avatar answered Sep 21 '22 06:09

anishsane


If you don’t want to create real files on your slow hard-disk, you can use FIFOs (First In First Out), which are also called named pipes, because of their behaviour.

mkfifo myfifo
generator | tee myfifo | processor | verifier myfifo

This streams the generated content to tee, which duplicates it to myfifo and to stdout, which is piped through the processor to the verifier. And the verifier also gets the stream from myfifo.

like image 22
erik Avatar answered Sep 22 '22 06:09

erik