Bash complex pipeline dependencies

Question

I'm trying to model a build concurrent pipeline in a single Bash script. I know I can use other tools, but at this point I'm doing it out of understanding Bash.

Scheduling jobs parallel is easy, and waiting for them all at the end is easy. But I want to make it run faster by trigger Task A.2 immediately after Task A.1 & Task X. To make it even hard on myself, the code in Task A.1 & Task A.2 is related & sequential, so it would be nice if I could keep the code sequential as well.

#!/usr/bin/bash

{
    echo "Task X"
} &
DEPENDENCY=$!

{
    echo "Task A.1"
    wait "${DEPENDENCY}"
    echo "Task A.2"
} &

{
    echo "Task B.1"
    wait "${DEPENDENCY}"
    echo "Task B.2"
} &

wait

This is ideally what I want, but it doesn't work because child processes can't wait for each other--which makes sense--but I'm hoping I can make this work in a cross platform way.

I actually have this working, but but I wasn't able to keep the code for part *.1 and *.2

Also it would be nice if this can work for OSX & Linux. I'm hoping a Bash expert can chime in and show a succinct way of expressing this in Bash.

Answer 1

As the comments to the question point out, the following Makefile is equivalent to your shell script (but doesn’t touch the file system)

.PHONY: a1 a2 b1 b2 x

all: a2 b2 

a1:
    @echo "Task A.1"

b1:
    @echo "Task B.1"

x:
    @sleep 1.5; echo "Task X"

a2: a1 x
    @echo "Task A.2"

b2: b1 x
    @echo "Task B.2"

Try it with make -j5 to allow for five parallel threads. It requires GNU make, which is available for every platform in existence, even if it might not be distributed by default. The initial .PHONY: target ensures that all tasks are executed even if files named a1, a2, ... happen to exist.

Answer 2

One way to make this work for OSX & Linux is to use namedpipes to signal to batch jobs that they can continue. Unfortunately it requires splitting up "Task *.1" & "Task *.2", but it's the best I can come up with for the moment.

#!/bin/bash

function cleanup {
  rm -f .task.a.1.done .task.b.1.done
}
trap cleanup EXIT
cleanup

mkfifo .task.a.1.done
mkfifo .task.b.1.done

{
    echo "Task A.1"
    echo -n "" > .task.a.1.done
} &

{
    echo "Task B.1"
    echo -n "" > .task.b.1.done
} &

{
    echo "Task X"
}

{
    cat < .task.a.1.done
    echo "Task A.2"
} &

{
    cat < .task.b.1.done
    echo "Task B.2"
} &

wait