GNU Parallel | pipe command

Question

I am completely new in using GNU parallel and I need your advice in running the command below using GNU parallel:

/home/admin/Gfinal/decoder/decdr.pl --gh --w14b /data/tmp/KRX12/a.bin | 
perl  /home/admin/decout/decoder/flow.pl >> /data/tmp/decodedgfile/out_1.txt

I will run this command on a list of files (.bin), so what is the best (fastest) approach to achieve that using GNU parallel noting that the output of the first part of the command (/home/admin/Gfinal/decoder/decdr.pl --gh --w14b) is very large (> 2 GB).

Any help would be appreciated.

Answer 1

Will this work:

parallel /home/admin/Gfinal/decoder/decdr.pl --gh --w14b {} '|' perl  /home/admin/decout/decoder/flow.pl >> /data/tmp/decodedgfile/out_1.txt ::: /data/tmp/KRX12/*.bin

(If the output from flow.pl is more than your disk I/O can handle, try parallel --compress).

Or maybe:

parallel /home/admin/Gfinal/decoder/decdr.pl --gh --w14b {} '|' perl  /home/admin/decout/decoder/flow.pl '>>' /data/tmp/decodedgfile/out_{#}.txt ::: /data/tmp/KRX12/*.bin

It depends on whether you want a single output file or one per input file.

Also spend an hour walking through the tutorial. Your command line will love you for it. man parallel_tutorial