I would like to use following for generating unique jobs, where {1} and {2} are unique tuples:
parallel echo {1} {2} ::: A B C D ::: A B C D
For instance in python (itertools) provides such a combinatoric generator:
permutations('ABCD', 2)
AB AC AD BA BC BD CA CB CD DA DB DC
Is there a way to implement it directly via bash? Or GNU parallel itself? Maybe skip redundant jobs somehow? But then, how can I check which parameter combinations were already used.
parallel echo {= 'if($_==3) { skip() }' =} ::: {1..5}
Below GNU parallel reads the last argument when generating the second job. When GNU parallel reads the last argument, it spreads all the arguments for the second job over 4 jobs instead, as 4 parallel jobs are requested.
For better parallelism GNU parallel can distribute the arguments between all the parallel jobs when end of file is met. Below GNU parallel reads the last argument when generating the second job. When GNU parallel reads the last argument, it spreads all the arguments for the second job over 4 jobs instead, as 4 parallel jobs are requested.
With a maximal line length of 10000 chars 17 commands will be run: Output: For better parallelism GNU parallel can distribute the arguments between all the parallel jobs when end of file is met. Below GNU parallel reads the last argument when generating the second job.
The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.
If the values are unique:
parallel echo {= 'if($arg[1] eq $arg[2]) { skip() }' =} ::: A B C D ::: A B C D
Or more generally:
parallel echo \
'{= my %seen; for my $a (@arg) { $seen{$a}++ and skip() } =}' \
::: A B C D ::: A B C D ::: A B C D
If you want to treat AB
as BA
then this only runs one of the combinations:
parallel echo \
'{= for my $t (2..$#arg) { if($arg[$t-1] ge $arg[$t]) { skip() } } =}' \
::: A B C D ::: A B C D ::: A B C D
If you use these alot, remember you can use --rpl
to make your own replacement strings by putting this in ~/.parallel/config
--rpl '{unique} my %seen; for my $a (@arg) { $seen{$a}++ and skip() }'
--rpl '{choose_k} for my $t (2..$#arg) { if($arg[$t-1] ge $arg[$t]) { skip() } }'
And then run:
parallel echo {unique} ::: A B C D ::: A B C D ::: A B C D
parallel echo {choose_k} ::: A B C D ::: A B C D ::: A B C D
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With