How to edit processes queued using && in Linux/bash?

Question

Lets say I have 3 processes queued in a Linux/Shell terminal by separating the commands for respective processes using && as follows:

<command1> && <command3> && <command4>

Now while command for process1 was being executed, I want to edit the queue:

---

e.g. 1: I want to have another process2 in between process1 and process3. So that the updated command queue becomes:

<command1> && <command2> && <command3> && <command4>

e.g. 2: Or may be remove a process for e.g. process3 from the queue. So that the updated command queue becomes:

<command1>  && <command4>

Is there a way to edit the command queue on the fly? i.e. when the process1 is already being executed?

Answer 1

You are talking about command list in bash (not queue), and once the shell parses it, you can not modify it.

It's not clear from the question whether you really need a queue, or a simple exit status testing will serve you, for example:

#!/bin/bash
if cmd1; then
    cmd2
else
    if cmd3; then
        cmd4
    fi
fi

(This would first run cmd1 and then, depending on its success, run cmd2 if it succeeded, or try running cmd3, followed with cmd4. The last if block can be simplified to cmd3 && cmd4.)

But, if you really need a command queue, you'll have to implement one yourself.

Try 1: simple array queue

The first idea is to store your queue in a shell array variable, like this:

#!/bin/bash

declare -a queue

clear() { queue=(); }

is_empty() (( ! ${#queue[@]} ))

# insert an item at the beginning
insert() {
    queue=("$1" "${queue[@]}")
}

# append an item at the end
push() {
    queue=("${queue[@]}" "$1")
}

# remove an item from the beginning
pop() {
    queue=("${queue[@]:1}")
}

# return the first item, without popping it
peek() {
    echo "${queue[0]}"
}

and then use it for running commands like this:

# run commands from the `queue`
run() {
    while ! is_empty; do
        local cmd=$(peek)
        pop
        eval "$cmd" || return
    done
    return 0
}

# run: echo before && sleep 1 && echo after
clear
push 'echo before'
push 'sleep 1'
push 'echo after'
run

But the major problem with this approach is you can't really modify this queue asynchronously. You can modify it before running, or while running inside the run loop, but that's probably not what you are asking for.

You might be thinking, why can't we execute commands in background with run &. Well, we could, but then the background subprocess (subshell) would receive its own copy of the queue variable, and the modifications you do after spawning would not be reflected in the run subprocess.

Try 2: simple file queue

Another approach is to implement your queue in a file, by storing one command per line. This would actually work, but we would also need to ensure some mutual exclusion mechanism is in place (for example flock).

While this approach works, I don't like the idea of rewriting the complete file each time you need to insert a command in position 0. You could keep the file in memory, though, by creating it on /dev/shm (ramdisk), but that wouldn't work on MacOS.

Try 3: Redis queue

Logical extension of the previous approach is to use an actual shared in-memory queue, like the one provided with Redis.

bash wrapper for this Redis-based queue would be:

#!/bin/bash

redis=(redis-cli --raw)

clear() {
    "${redis[@]}" DEL queue &>/dev/null
}

is_empty() (( $("${redis[@]}" LLEN queue) == 0 ))

# insert the item at the beginning
insert() {
    "${redis[@]}" LPUSH queue "$1" &>/dev/null
}

# append the item at the end
push() {
    "${redis[@]}" RPUSH queue "$1" &>/dev/null
}

# remove the item from the beginning
pop() {
    "${redis[@]}" LPOP queue
}

peek() {
    "${redis[@]}" LRANGE queue 0 0
}

show() {
    "${redis[@]}" LRANGE queue 0 -1
}

And you could run your first example (with insertion while the previous command is running), like this:

# run commands from the redis queue
run() {
    while ! is_empty; do
        eval "$(pop)" || return
    done
    return 0
}

# start with: echo before && sleep 1 && echo after
clear
push 'echo before'
push 'sleep 3'
push 'echo after'

run &

# but then, 1sec after running, modify the queue, insert another command
sleep 1
insert 'echo inserted'

wait

Example output:

$ ./redis-queue-demo.sh
before
inserted
after

---

The solution for your examples

So, using the Redis approach, your first example (inserting a command) would look like:

clear
push 'echo command1; sleep 2'
push 'echo command3'
push 'echo command4'

run &

sleep 1
insert 'echo command2'

wait

Output:

command1
command2
command3
command4

The second example (removing a command):

clear
push 'echo command1; sleep 2'
push 'echo command3'
push 'echo command4'

run &

sleep 1
pop >/dev/null

wait

outputs:

command1
command4