Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

GNU parallel not spawning jobs

After an upgrade to Debian 8.6 Jessie the GNU parallel script suddenly stopped parallelizing to more than 2 jobs with the --pipe and -L options.

Before the upgrade the command:

cat file_with_1064_lines.txt | parallel -L10 -j5 -k -v --pipe "wc -l"

spawned 5 processes, which output this:

wc -l
10
wc -l
10
...

The same command after the upgrade:

wc -l
1060
wc -l
4

(The two values above change with respect to the -L option value -- the first is L*floor(1064/L) and the second is 1064 mod L, but there always only two processes outputting.)

The same is observed independently of the parallel version (tested the latest and one from 2013).

PS.

$ uname -a

Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux

$ parallel --version

GNU parallel 20161222
like image 501
artem Avatar asked Dec 05 '25 10:12

artem


1 Answers

-L is the record size. The bug was fixed around 20130122. What you want is to read 1 record of 10 lines:

parallel -L10 -N1 -j5 -k -v --pipe wc -l

or 10 records of 1 line:

parallel -L1 -N10 -j5 -k -v --pipe wc -l
like image 129
Ole Tange Avatar answered Dec 07 '25 02:12

Ole Tange



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!