Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to make pipe run sequentially

Specs

Don't think this should make a difference but I'll include it anyway

GNU bash, version 3.2.51


Premise

So say i have a pipe with multiple parts, how can I prevent part of the pipe running before the previous part is finished.


Example

In the below examples I will try to display the issue

$ echo hello | tee /dev/tty | (echo "Next";sed 's/h/m/' )

Output

Next
hello
mello

With sleep to show that it is the timing that is off

$ echo hello | tee /dev/tty | (sleep 2;echo "Next";sed 's/h/m/' )

Output

hello
Next
mello

Expected Output

Is as above

hello
Next
mello

but obviously this depends on the sleep being longer than it takes for the previous command to finish which is not what i would want.


Notes

I know there are better ways to do this but I just think it would be educational for me to understand exactly how the pipe is working.


Tried

Tried variations of wait and sleep and things but nothing works consistently.


l0b0's suggestion

This still prints Next first though

$ echo hello | tee /dev/tty | sort |(echo "Next";sed 's/h/m/' )
Next
hello
mello

$ echo hello | tee /dev/tty | tac | tac |(echo "Next";sed 's/h/m/' )
Next
hello
mello

If any more information is needed then please let me know.

like image 904
123 Avatar asked Aug 21 '15 08:08

123


2 Answers

The point of a pipe is to process data asynchronously, so as to save time and space overall. If you want to have a synchronous pipe you might as well write to file (on a RAM disk if you need the speed). But for tasks where the receiving commands are able to handle data in chunks the complete pipeline may be much slower:

  1. a | b | c can at best be as fast as the slowest of the three commands.
  2. a > file; b < file > file2; c < file2 can at best be as fast as the sum of the runtimes of each command.

So if the commands all run in about N seconds (when run separately), you're looking at a best case runtime of N for the first command and 3N for the second command.

like image 191
l0b0 Avatar answered Sep 21 '22 22:09

l0b0


There is no language construct in bash to modify the behavior of a pipeline like you want. However, you can use a named pipe to act as a type of binary semaphore:

mkfifo block
echo hello | 
  { tee /dev/tty; echo go > block; }  |
  (read < block; echo "Next"; sed 's/h/m/' )

The read command blocks until something writes to named pipe, which does't happen until tee completes.

(Note that this may not completely solve your problem, as in addition to the process synchronization you may need to contend with the fact that you have multiple processes writing to the same output file, and you don't have complete control over how the various writes are multiplexed (due to buffering, etc.)

like image 34
chepner Avatar answered Sep 20 '22 22:09

chepner