Looking for a scalable "at" implementation

Question

I'm looking for a scalable "at" replacement, with high availability. It must support adding and removing jobs at runtime.

Some background: I have an application where I trigger millions of events, each event occurs just once. I don't need cron like mechanisms (first Sunday of the month, etc), simply date, time and context.

Currently I'm using the Quartz scheduler, and while it is a very good project, it has difficulties to handle the amount of events we throw at it, even after a lot of tweaking (sharding, increasing polling interval, etc.) due to the basic locking it performs on the underline database. Also, it is a bit overkill for us, as basically we have millions of one time triggers, and relatively small number of jobs.

I'd appreciate any suggestion

Answer 1

If I was facing the same scenario I would do the following...

Setup a JMS queue cluster (e.g RabbitMQ or ActiveMQ) using a queue replication setup over a few boxes.

Fire all the events at my nice highly-available JMS queue.

Then I would code an agent application that popped the events of the JMS queue as needed, I could run multiple agents on multiple boxes and have that combined with the correct JMS failover url etc.

You could also use the same sort of model if your jobs are firing the events...

Fyi, a nicer way of scheduling in core Java is as follows:

ScheduledExecutorService threadPool = Executors.newScheduledThreadPool(sensibleThreadCount);
    threadPool.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
       //Pop events from event queue.
       //Do some stuff with them.
    }

    }, initialDelay, period, TimeUnit.X);