Persistent job queue?

Question

Internet says using database for queues is an anti-pattern, and you should use (RabbitMQ or Beanstalked or etc)

But I want all requests stored. So I can later lookup how long they took, any failed attempts or errors or notes logged, who requested it and with what metadata, what was the end result, etc.

It looks like all the queue libraries don't have this option. You can't persist the data to allow you to query it later.

I want what those queues do, but with a "persist to database" option. Does this not exist? How do people deal with this? Do you use a queue library and copy over all request information into your database when the request finishes?

(the language/database I'm using is anything, whatever works best for this)

Answer 1

If you want to log requests, and meta-data about how long they took etc, then do so - log it to the database when you know the relevant results, and run your analytic queries as you would expect to.

The reason to not be using the database as a temporary store is that under high traffic, the searching for, and locking of unprocessed jobs, and then updating or deleting them when they are complete, can take a great deal of effort. That is especially true if don't remove jobs from the active table, and so have to search ever more completed jobs to find those that have yet to be done.