Menu
GNU parallel examples to run command or code in parallel in bash shell. From the GNU project site: GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input.
In case you need to execute several processes in batches, or in chunks, you can use the shell builtin command called "wait". See below. The first three commands wget commands will be executed in parallel. "wait" will make the script wait till those 3 gets finished.
When you execute a Bash script, it will at maximum use a single CPU thread, unless you start subshells/threads. If your machine has at least two CPU threads, you will be able to max-out CPU resources using multi-threaded scripting in Bash.
Use the wait built-in:
process1 &
process2 &
process3 &
process4 &
wait
process5 &
process6 &
process7 &
process8 &
wait
For the above example, 4 processes process1 ... process4 would be started in the background, and the shell would wait until those are completed before starting the next set.
From the GNU manual:
wait [jobspec or pid ...]Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.
See parallel. Its syntax is similar to xargs, but it runs the commands in parallel.
In fact, xargs can run commands in parallel for you. There is a special -P max_procs command-line option for that. See man xargs.
You can run 20 processes and use the command:
wait
Your script will wait and continue when all your background jobs are finished.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With