Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Exporting the full environment to GNU Parallel

I find it somewhat annoying that I cannot use aliases in GNU Parallel:

alias gi="grep -i"
parallel gi bar ::: foo
/bin/bash: gi: command not found

I had somewhat come to terms with that it is just the way it is. But reading Accessing Associative Arrays in GNU Parallel I am starting to think: Does it really have to be this way?

Is is possible to make a bash function, that collects all of the environment into a function, exports that function and calls GNU Parallel, which will then import the environment in the spawned shell using that function?

So I am not talking about a specialized solution for the gi-alias, but a bash function that will take all aliases/functions/variables (without me having to name them explicitly), package those into a function, that can be activated by GNU Parallel.

Something similar to:

env_parallel() {
  # [... gather all environment/all aliases/all functions into parallel_environment() ...]
  foreach alias in all aliases {
     append alias definition to definition of parallel_environment()
  }
  foreach variable in all variables (including assoc arrays) {
     append variable definition to definition of parallel_environment()
     # Code somewhat similar to https://stackoverflow.com/questions/24977782/accessing-associative-arrays-in-gnu-parallel
  }
  foreach function in all functions {
     append function definition to definition of parallel_environment()
  }

  # make parallel_environment visible to GNU Parallel
  export -f parallel_environment

  # Running parallel_environment will now create an environment with
  # all variables/all aliases/all functions set in current state 
  # (with the exception of the function parallel_environment of course)

  # Inside GNU parallel:
  #    if set parallel_environment(): prepend it to the command to run
  `which parallel` "$@"
}

# Set an example alias
alias fb="echo fubar"
# Set an example variable
BAZ=quux
# Make an example function
myfunc() {
  echo $BAZ
}

# This will record the current environment including the 3 examples
# put it into parallel_environment
# run parallel_environment (to set the environment)
# use the 3 examples
env_parallel parallel_environment\; fb bar {}\; myfunc ::: foo

# It should give the same output as running:
fb bar foo; myfunc
# Outputs:
#   fubar bar foo
#   quux

Progress: This seems to be close to what I want activated:

env_parallel() {
  export parallel_environment='() {
    '"$(echo "shopt -s expand_aliases"; alias;typeset -p | grep -vFf <(readonly);typeset -f)"'
  }'
  `which parallel` "$@"
}

VAR=foo
myfunc() {
  echo $VAR $1
}
alias myf=myfunc
env_parallel parallel_environment';
' myfunc ::: bar # Works (but gives errors)
env_parallel parallel_environment';
' myf ::: bar # Works, but requires the \n after ;

So now I am down to 1 issue:

  • weed out all the variables that cannot be assigned value (e.g BASH_ARGC)

How do I list those?

like image 879
Ole Tange Avatar asked Sep 19 '25 12:09

Ole Tange


1 Answers

GNU Parallel 20140822 implements this. To activate it you will need to run this once (e.g. in .bashrc):

env_parallel() {
    export parallel_bash_environment='() {
       '"$(echo "shopt -s expand_aliases 2>/dev/null"; alias;typeset -p | grep -vFf <(readonly; echo GROUPS; echo FUNCNAME; echo DIRSTACK; echo _; echo PIPESTATUS; echo USERNAME) | grep -v BASH_;typeset -f)"'
       }'
     # Run as: env_parallel ...
     `which parallel` "$@"
     unset parallel_bash_environment
}

And call GNU Parallel as:

env_parallel ...

That should put the myth to rest that it is impossible to export aliases: all you need is a little Behändigkeit (Thanks a lot to @rici for the inspiration).

like image 175
Ole Tange Avatar answered Sep 22 '25 06:09

Ole Tange