Job Dependency in RabbitMQ

Question

I am trying to figure out how to best setup the following scenario:

1. Multiple A-type jobs are added into the queue
2. When all A-type jobs are completed, a B or C-type job will be needed (one per A-type job)
3. When all A, B and C type jobs are completed, a final D-type job will be needed

So basically we have some dependencies on jobs in the queue such that we don't want to start running jobs that require other jobs to be completed. Is there a guideline for setting up such a system? Should A-type jobs add B or C type jobs after their work is completed? Should all jobs be added up front and somehow tell the workers not to pull them until they are ready?

There are pros and cons of both approaches if I have to manually manage this dependency but I am curious if there is a different pattern I could use instead that might accomplish the same thing but in an easier way.

Answer 1

I'm wondering if the routing slip pattern could inspire you here, ie send in the original A messages a "slip" that defines what the next messages (B, C) should be (what rules to apply to figure it out at runtime).

I would also add in the slip a unique correlation ID and the correlation group size (in that case: the total number of A messages).

For the final D job, I'd have the B/C process steps send a "D-ready" request message. When all the D-ready messages would have been collected (based on correlation ID/group size), I would trigger the final D job.