Unique ids for jobs in a message queue?

Question

I have an application which I'm writing which needs to perform long computations in the background, so I have essentially the following workflow:

1. Client submits job to an edge "dispatcher" server.
2. Dispatcher server submits job to message queue.
3. Compute server pulls job and starts work.

The compute server also provides live feedback on the status of the work, so as to make it possible for clients to watch progress.

The main problem I'm having right now is figuring out how to get a unique job id for a submitted message in the queue, and also to figure out after the fact which server serviced the message. Once the job is initially submitted (step 1), the client should receive a unique token identifying the job. The client should then be able to periodically poll the dispatcher server to check the status of the token on whether it's been started or not.

After a compute server has serviced the request, the client should then get the DNS address or IP address of the encoder server in the poll call.

How can I make this happen? Do message queues provide this notion of unique identifying tokens for each message in the queue?

Answer 1

As it turns out, RabbitMQ does in fact give unique ids for messages, especially useful in RPC applications, which is the kind of application I'm looking at creating. Each message has a correlationId which uniquely identifies it and can provide a callback by which to inform the original server when the job has completed or really at any point at all.