Decouple programs using queues

Question

In his talk at the 54:53 minute mark, Rich Hickey is talking about the usage of queues as a mean to decouple dependent program parts. Can you give me an example on how to deouple the following piece of Java-pseudo-code in order to improve it's design and/or flexibility:

// Warning: Java-pseudo-code ahead
class Job {
    public void doRun(A a) {
        saveObjectToDatabase(a);

        B b = computeB(a);
        saveObjectToDatabase(b);

        C c = computeC(b);
        logToFile(c);
    }
}

saveObjectToDatabase and saveObjectToDatabase can be seen as a method with side-effects, whereas computeB's and computeC's output only depend on a.

I know this question is rather vague/broad. I would like to get a feeling on how to leverage queueing mechanisms without massively complicating my program and still making sure it does the right thing in the right order. Any pointers into the right direction are appreciated.

Answer 1

Well, it's not a very good example, but (in the most straight-forward design) you'd basically have two queues, and (depending on the amount of data involved) you might omit the database.

A first process would receive your a objects from the "outside world" and enqueue them in queue 1. A second process would dequeue objects from queue 1, perform computeB, and enqueue the results onto queue 2. A third process would dequeue objects from queue 2, perform computeC, and log the result or whatever.

Depending, as I said, on the amount of data involved (and maybe a few other factors) the "objects" passed in the queues could either be your actual a and b objects or else just tokens/keys to find the data in the database.

The queues themselves could be implemented several ways. It's possible to implement a queue with a database, eg, though the details get kind of messy. The "processes" could be Java tasks within a single Java process or could be separate OS processes, possibly even on separate machines.

When you use "pipes" on Unix you are effectively using queues in this fashion.