Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Add a big buffer to a pipe between two commands

Tags:

bash

unix

Given a bash command line of the form

commandA | commandB 

I want to add a buffer of size ~1MB that sits between commandA and commandB. I would expect to be able to do this with something of the form

commandA | BUFFER | commandB 

but what is the command to use for BUFFER?

Remark: I want to do this in order to decouple the two commands to make them parallelize better. The problem is that commandB processes data in large chunks, which currently means that commandA blocks until commandB is done with a chunk. So everything runs sequentially :-(

like image 869
Alex Krauss Avatar asked Dec 18 '11 20:12

Alex Krauss


People also ask

How can we connect more than one command using pipe?

Use parentheses ()'s to combine the commands into a single process, which will concatenate the stdout of each of them.

Is a pipe a buffer?

PIPES is the common name for piperazine-N,N′-bis(2-ethanesulfonic acid), and is a frequently used buffering agent in biochemistry. It is an ethanesulfonic acid buffer developed by Good et al. in the 1960s.


1 Answers

BUFFER is called buffer. (man 1 buffer, maybe after apt-get install buffer)

like image 107
Eugen Rieck Avatar answered Oct 14 '22 18:10

Eugen Rieck