Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

GNU parallel combinatorics, usage of list of argument multiple times

I would like to use following for generating unique jobs, where {1} and {2} are unique tuples:

parallel echo {1} {2} ::: A B C D ::: A B C D

For instance in python (itertools) provides such a combinatoric generator:

permutations('ABCD', 2)

AB AC AD BA BC BD CA CB CD DA DB DC


Is there a way to implement it directly via bash? Or GNU parallel itself? Maybe skip redundant jobs somehow? But then, how can I check which parameter combinations were already used.

parallel echo {= 'if($_==3) { skip() }' =} ::: {1..5}
like image 853
mdavies Avatar asked Sep 19 '17 10:09

mdavies


People also ask

Does GNU parallel read the last argument when generating the second job?

Below GNU parallel reads the last argument when generating the second job. When GNU parallel reads the last argument, it spreads all the arguments for the second job over 4 jobs instead, as 4 parallel jobs are requested.

How to distribute arguments between all the parallel jobs?

For better parallelism GNU parallel can distribute the arguments between all the parallel jobs when end of file is met. Below GNU parallel reads the last argument when generating the second job. When GNU parallel reads the last argument, it spreads all the arguments for the second job over 4 jobs instead, as 4 parallel jobs are requested.

How many commands will be run in GNU parallel?

With a maximal line length of 10000 chars 17 commands will be run: Output: For better parallelism GNU parallel can distribute the arguments between all the parallel jobs when end of file is met. Below GNU parallel reads the last argument when generating the second job.

What is the typical input of GNU parallel?

The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.


1 Answers

If the values are unique:

parallel echo {= 'if($arg[1] eq $arg[2]) { skip() }' =} ::: A B C D ::: A B C D

Or more generally:

parallel echo \
  '{= my %seen; for my $a (@arg) { $seen{$a}++ and skip() } =}' \
  ::: A B C D ::: A B C D ::: A B C D

If you want to treat AB as BA then this only runs one of the combinations:

parallel echo \
  '{= for my $t (2..$#arg) { if($arg[$t-1] ge $arg[$t]) { skip() } } =}' \
  ::: A B C D ::: A B C D ::: A B C D

If you use these alot, remember you can use --rpl to make your own replacement strings by putting this in ~/.parallel/config

--rpl '{unique} my %seen; for my $a (@arg) { $seen{$a}++ and skip() }'
--rpl '{choose_k} for my $t (2..$#arg) { if($arg[$t-1] ge $arg[$t]) { skip() } }'

And then run:

parallel echo {unique} ::: A B C D ::: A B C D ::: A B C D
parallel echo {choose_k} ::: A B C D ::: A B C D ::: A B C D
like image 119
Ole Tange Avatar answered Nov 15 '22 08:11

Ole Tange