GNU Parallel in BASH script with "export -f <func>" failed with "Command Not Found" error when Crond

Question

My script works if I run it interactively on command shell:

$ cat ndmpcopy_cron_parallel_svlinf05.bash
#!/usr/software/bin/bash

ndmpcopy_cron_parallel() {

timestamp=`date +%Y%m%d-%H%M`
LOG=/x/eng/itarchives/ndmpcopylogs/05_$1/ndmpcopy_status
TSLOG=${LOG}_$timestamp

src_filer='svlinf05'
src_account='ndmp'
src_passwd='src_passwd'
dst_svm='svlinfsrc'
dst_account='vsadmin-backup'
dst_passwd='dst_passwd'

host=`hostname`
echo $host

ssh -l root $src_filer "priv set -q diag ; ndmpcopy -sa $src_account:$src_passwd -da $dst_account:$dst_passwd -i $src_filer.eng.netapp.com:/vol/$1 10.56.10.161:/$dst_svm/$1" | tee -a $TSLOG

echo "ndmpcopy Completed: `date` "

}

export -f ndmpcopy_cron_parallel

/u/jsung/bin/parallel -j 0 --wd . --env ndmpcopy_cron_parallel --eta ndmpcopy_cron_parallel ::: local

But, the script failed and complained the exported function, ndmpcopy_cron_parallel, cannot be found:

$ crontab -l

40 0,2,4,6,8,10,12,14,16,18,20,22 * * * /u/jsung/bin/ndmpcopy_cron_parallel_svlinf05.bash

Error:

Subject: Cron <jsung@cycrh6svl18> /u/jsung/bin/ndmpcopy_cron_parallel_svlinf05.bash


Computers / CPU cores / Max jobs to run
1:local / 2 / 1

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
 ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s **/bin/bash: ndmpcopy_cron_parallel: command not found**
 ETA: 0s Left: 0 AVG: 0.00s  local:0/1/100%/0.0s 

I've been searched around and trying different things for a while. I even tweaked $PATH. Not sure what I missed. Can we embed GNU Parallel in BASH script and put in crontab at all?

Answer 1

Congratulations. You've been shell-shocked.

You have two versions of bash installed on your system:

• /bin/bash v4.1.2 An older unpatched bash
• /usr/software/bin/bash v4.2.53 A middle-aged bash, patched against Shellshock

The last number in the bash version triple is the patch-level. The Shellshock bug involved a number of patches, but the relevant one is 4.1.14, 4.2.50 and 4.3.27. That patch changes the format of exported functions, with the consequence that:

• If you export a function from a pre-shellshock bash to a post-shellshock bash, you will see a warning and the exported function will be rejected.
• If you export a function from a post-shellshock bash to a pre-shellshock bash, the function export format won't be recognized so it will be silently ignored.

In both cases, the function will not be exported. In other words, you can only export a function between two bash versions if they have both been shellshock patched, or if neither have been shellshock patched.

Your script clearly indicates which bash to use to run it: the one in /usr/software/bin/bash, which has been patched. The script invokes GNU parallel, and GNU parallel then has to start up one or more subshells in order to run the commands. GNU parallel uses the value of the SHELL environment variable to find the shell it should use.

I suppose that in your user command shell environment, SHELL is set to /usr/software/bin/bash, and that in the environment in which cron executes, it is set to /bin/bash. If that's the case, you'll have no problems exporting the function when you try it from a bash prompt, but in the cron environment you will end up trying to export a function from a post-shellshock bash to a pre-shellshock bash, and as described above the result is that the export is silently ignored. Hence the error.

To get around the problem, you need to ensure that you use the bash used to run the command script is the same as the bash used by GNU parallel. You could, for example, explicitly set the shell prior to invoking GNU parallel.

export SHELL=/usr/software/bin/bash
# ...
/u/jsung/bin/parallel -j 0 --wd . --env ndmpcopy_cron_parallel --eta ndmpcopy_cron_parallel ::: local

Or you could just set it for the parallel command itself:

SHELL=/usr/software/bin/bash /u/jsung/bin/parallel -j 0 --wd . --env ndmpcopy_cron_parallel --eta ndmpcopy_cron_parallel ::: local

Answer 2

As rici says, the problem is most likely due to shellshock. Shellshock did not affect GNU Parallel, but the patches to fix shellshock broke transferring of functions using '--env'.

GNU Parallel is catching up with the shellshock patches in Bash: Bash has used BASH_FUNC_myfunc() as the variable name for exporting functions, but more recent versions use BASH_FUNC_myfunc%%. So GNU Parallel needs to know this when transferring a function.

The '()' version is fixed in 20141022, and the '%%' version is expected to be fixed in 20150122. They should work in any combination. So your remote Bash does not need to be patched the same way as your local Bash: GNU Parallel will "do the right thing", and there is no need to change your own code.

You should feel free to test out the git version in which both are fixed: git clone git://git.savannah.gnu.org/parallel.git