Persistent delayed jobs queue for Java

Question

I'm looking for an existing system to replace an existing slow and complicated self written mechanism of job management. The existing system:

• 1 MySQL DB with a long massive table of jobs - the queue
• Multiple servers (written in java) all extracting jobs from the queue and processing them
• a job might NOT be deleted from the queue after processing it, to rerun it later
• a job might create other jobs and insert them to the queue

The limitations:

• As more and more jobs are created and inserted in to the queue, it takes longer to extract jobs from it. (The jobs are chosen by priority and type) - create a bottle neck

I'm looking for an existing system that can replace this one, and improve it's performance. Any suggestions?

Thanks

Answer 1

I don't generally recommend JMS, but it sounds like it really is what you need here. Distributed, transactional, persistent job queue management is what JMS is all about.

Popular open-source implementations include HornetQ and ActiveMQ.