After an upgrade to Debian 8.6 Jessie the GNU parallel script suddenly stopped parallelizing to more than 2 jobs with the --pipe and -L options.
Before the upgrade the command:
cat file_with_1064_lines.txt | parallel -L10 -j5 -k -v --pipe "wc -l"
spawned 5 processes, which output this:
wc -l 10 wc -l 10 ...
The same command after the upgrade:
wc -l 1060 wc -l 4
(The two values above change with respect to the -L option value -- the first is L*floor(1064/L) and the second is 1064 mod L, but there always only two processes outputting.)
The same is observed independently of the parallel version (tested the latest and one from 2013).
PS.
$ uname -a Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux $ parallel --version GNU parallel 20161222
-L is the record size. The bug was fixed around 20130122. What you want is to read 1 record of 10 lines:
parallel -L10 -N1 -j5 -k -v --pipe wc -l
or 10 records of 1 line:
parallel -L1 -N10 -j5 -k -v --pipe wc -l
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With