Queue synchronization

Question

I am working with Java rabbitmq API. This how my app looks like:

Publisher A sends jobs to Q1, then many consumers B do the job and create new job and send it to Q2, where many consumer C do the job.

I want to make sure that no duplicated jobs are sent to Q2, how can I achieve this?

Take in mind two scenarios:

1. B crash after sending job to Q2 but before sending acknowledgment to Q1 that he completed the job
2. B crash after sending acknowledgment to Q1 but before sending job to Q2

Answer 1

I want to make sure that no duplicated jobs are sent to Q2, how can I achieve this?

you can't. not even on Q1.

the nature of distributed systems and the CAP theorem (https://en.wikipedia.org/wiki/CAP_theorem) says this is impossible, even if the goal is accomplished the vast majority of the time.

in light of that, what you need do is plan for how you will handle the times when a duplicate message is accidentally created.

the most common method is some sort of idempotence (https://en.wikipedia.org/wiki/Idempotence) - a way to guarantee that the same message will only be processed once. or, maybe more accurately, a way to say that the same message can be processed an unlimited number of times, but will only cause change / have an effect on the system once.

---

in general, though, your situation with multiple queues and consumers that need to process things in order is calling for a "Saga" or "Process Manager" - a long running, asynchronous workflow.

You can read up on this idea of a "Process Manager" in the Enterprise Integration Patterns book, and there are a lot of good libraries around that will implement the details for you.

• http://www.enterpriseintegrationpatterns.com/patterns/messaging/ProcessManager.html

• http://kellabyte.com/2012/05/30/clarifying-the-saga-pattern/