Is it possible to distribute STDIN over parallel processes?

Question

Given the following example input on STDIN:

foo
bar bar
baz
===
qux
bla
===
def
zzz yyy

Is it possible to split it on the delimiter (in this case '===') and feed it over stdin to a command running in parallel?

So the example input above would result in 3 parallel processes (for example a command called do.sh) where each instance received a part of the data on STDIN, like this:

do.sh (instance 1) receives this over STDIN:

foo
bar bar
baz

do.sh (instance 2) receives this over STDIN:

qux
bla

do.sh (instance 3) receives this over STDIN:

def
zzz yyy

I suppose something like this is possible using xargs or GNU parallel, but I do not know how.

Answer 1

In general, no. One of the reasons for this assessment is that standard I/O reading from files, rather than the terminal, reads blocks of data - BUFSIZ bytes at a time, where BUFSIZ is usually a power of 2 such as 512 or larger. If the data is in a file, one process would read the whole file shown - the others would see nothing if they shared the same open file description (similar to a file descriptor, but several file descriptors can share the same open file description, and could be in different processes), or would read the whole same file if they did not share the same open file description.

So, you need a process to read the file that knows it needs to parcel the information out to the three processes - and it needs to know how it is to connect to the three processes. It might be that your distributor program runs the three processes and writes to their separate pipe inputs. Or it could be that the distributor connects to three sockets and writes to the different sockets.

Your example doesn't show/describe what would happen if there were 37 sections separated by the marker.

I have a home-brew program called tpipe that is like the Unix tee command, but it writes a copy of (all of) its standard input to each of the processes, and to standard output too by default. This might be a suitable basis for what you need (it at least covers the process management part of it). Contact me if you want a copy - see my profile.

---

If you are using Bash, you can use regular tee with process substitution to simulate tpipe. See this article for an illustration of how.

See also SF 96245 for another version of the same information - plus a link to a program called pee that is quite similar to tpipe (same basic idea, slightly different implementation in various respects).

Answer 2

GNU Parallel can do that from version 20110205.

cat | parallel --pipe --recend '===\n' --rrs do_stuff