Menu
I want to design a job scheduler cluster, which contains several hosts to do cron job scheduling. For example, a job which needs run every 5 minutes is submitted to the cluster, the cluster should point out which host to fire next run, making sure:
- Disaster tolerance: if not all of the hosts are down, the job should be fired successfully.
- Validity: only one host to fire next job run.
Due to disaster tolerance, job cannot bind to a specific host. One way is all the hosts polling a DB table(certainly with lock), this guaranteed only one host gets the next job run. Since it often locks table, is there any better design?
773
Durability: Jobs must not get lost -> we need to persist jobs. Reliability: Jobs must not be executed much later than expected or dropped -> we need a fault-tolerant system. Availability: It should always be possible to schedule and execute jobs -> (dynamical) horizontal scaling.
For example, you may use source control to manage configuration data that you want to deploy to production daily or weekly. However, instead of having systems “pull” these configurations from source control with locally scheduled cron jobs, you can use Jenkins to act as a central scheduler.
Dkron is a system service for workload automation that runs scheduled jobs, just like unix cron service but distributed in several machines in a cluster. This is the only job scheduler in the market with truly no SPOF. It is open source and available for free.
Use the Quartz framework for that. It has a cron like syntax, can be clustered and only one of the hosts in the cluster will do one job at a time. If a host or job fails, another host will retry the pending job.
184
I googled out the Dkron (Distributed job scheduling system). It has rest api and looks good. I plan try to use it Dkron site
28
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
