Restarting quartz scheduler without getting an error

Question

Context

I am trying to use the quartz scheduler in cluster mode using jdbc.

Problem

Before I started with jdbc in clustered mode I just tested the scheduler in general with the RAM store. That worked without a problem and I was able to restart the scheduler (main class) without any errors. The problem I have now is that when I stop the execution (ctrl+c) and then restart it I always get the error message:

org.quartz.ObjectAlreadyExistsException: Unable to store Job : 'MyTestJob', because one already exists with this identification.

I don't understand what is going on here. Does quartz not support restarting the scheduler? I mean, what happens if there is a crash and the scheduler restarts after recovery? Is the only option to then delete the jobs from the quartz database? Perhaps there is another method or something that I have missed. I don't feel very comfortable using a library that does not cope with restarts.

Another odd thing is, that when changing to jdbc my job does not get triggered anymore and I just see the state WAITING in the DB. What could this be? The job (cron-schedule) worked without a problem in RAM mode.

I am a bit surprised about the level of documentation and the problems I am encountering with this simple task because I have heard of the quartz scheduler for many years now, but never got round to using it. Goodle suggests that I am not the only one with this problem. I hope that this is just me and that there is a simple solution to my problem, otherwise it would be very disappointing to try this library out for the first time in the 2.2.x version and already having to look for something else.

Here is my configuration:

#============================================================================
# Configure Main Scheduler Properties  
#============================================================================

org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = Test-Scheduler
org.quartz.scheduler.instanceId = AUTO

#============================================================================
# Configure ThreadPool  
#============================================================================

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 25
org.quartz.threadPool.threadPriority = 5

#============================================================================
# Configure JobStore  
#============================================================================

org.quartz.jobStore.misfireThreshold = 60000

org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = true
org.quartz.jobStore.dataSource = quartzDS
org.quartz.jobStore.tablePrefix = QRTZ_

org.quartz.jobStore.isClustered = true
org.quartz.jobStore.clusterCheckinInterval = 20000

Here is my code:

    SchedulerFactory sf = new StdSchedulerFactory();
    Scheduler scheduler = sf.getScheduler();

    scheduler.start();

                JobDetail jobDetail = newJob(job.getClass())
                    .withIdentity("test-name", "test-group")
                    .build();

                CronTrigger trigger = newTrigger()
                    .withIdentity("test-name-trigger", "test-group")
                    .withSchedule(cronSchedule("0 0/1 * * * ?"))
                    .build();

                scheduler.scheduleJob(jobDetail, trigger);

                System.out.println(trigger.getNextFireTime());

EDIT

This is interesting.

1) RAM mode works.

2) jdbc with cluster enable does not work and fails (almost) silently - even with logging enabled. In log output I see the following:

19:57:29,913  INFO StdSchedulerFactory:1184 - Using default implementation for ThreadExecutor
19:57:29,936  INFO SchedulerSignalerImpl:61 - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
19:57:29,936  INFO QuartzScheduler:240 - Quartz Scheduler v.2.2.1 created.
19:57:29,938  INFO JobStoreTX:667 - Using db table-based data access locking (synchronization).
19:57:29,940  INFO JobStoreTX:59 - JobStoreTX initialized.
19:57:29,941  INFO QuartzScheduler:305 - Scheduler meta-data: Quartz Scheduler (v2.2.1) 'Test-Scheduler' with instanceId 'Michael-PC1405447049916'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 25 threads.
  Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.

19:57:29,941  INFO StdSchedulerFactory:1339 - Quartz scheduler 'Test-Scheduler' initialized from default resource file in Quartz package: 'quartz.properties'
19:57:29,941  INFO StdSchedulerFactory:1343 - Quartz scheduler version: 2.2.1
19:57:29,995  INFO AbstractPoolBackedDataSource:462 - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1hgeby993gf1xpdmdc44s|7ec4d0, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hgeby993gf1xpdmdc44s|7ec4d0, idleConnectionTestPeriod -> 50, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://localhost:3306/scheduler, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 5, maxStatements -> 0, maxStatementsPerConnection -> 120, minPoolSize -> 1, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> SELECT 1 FROM QRTZ_JOB_DETAILS, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> true, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]
19:57:30,243 DEBUG StdRowLockSemaphore:107 - Lock 'TRIGGER_ACCESS' is desired by: main
19:57:30,262 DEBUG StdRowLockSemaphore:92 - Lock 'TRIGGER_ACCESS' is being obtained: main
19:58:21,328 DEBUG StdRowLockSemaphore:141 - Lock 'TRIGGER_ACCESS' was not obtained by: main - will try again.
19:58:22,329 DEBUG StdRowLockSemaphore:92 - Lock 'TRIGGER_ACCESS' is being obtained: main
19:59:13,389 DEBUG StdRowLockSemaphore:141 - Lock 'TRIGGER_ACCESS' was not obtained by: main - will try again.
19:59:14,389 DEBUG StdRowLockSemaphore:92 - Lock 'TRIGGER_ACCESS' is being obtained: main

Although, just as as I was about to enable cluster mode again, I saw the exception:

Exception in thread "main" org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: Lock wait timeout exceeded; try restarting transaction [See nested exception: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction]
    at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:157)
    at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:113)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3780)
    at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:93)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.clearAllSchedulingData(JobStoreSupport.java:1956)
    at org.quartz.core.QuartzScheduler.clear(QuartzScheduler.java:1572)
    at org.quartz.impl.StdScheduler.clear(StdScheduler.java:239)
    at com.scs.core.cron.TaskRunner.main(TaskRunner.java:52)

3) In jdbc mode with clustering disabled it does not work either, but I get an exception:

  20:04:15,993 DEBUG SimpleSemaphore:132 - Lock 'TRIGGER_ACCESS' retuned by: main
  20:04:15,993 DEBUG JobStoreTX:703 - JobStore background threads started (as scheduler was started).
  20:04:15,994  INFO QuartzScheduler:575 - Scheduler Test-Scheduler_$_NON_CLUSTERED started.
  20:04:15,994 DEBUG JobStoreTX:3933 - MisfireHandler: scanning for misfires...
  20:04:16,000 DEBUG JobStoreTX:3182 - Found 0 triggers that missed their scheduled fire-time.
  20:04:16,004 DEBUG QuartzSchedulerThread:276 - batch acquisition of 0 triggers
  20:04:16,008 DEBUG SimpleSemaphore:81 - Lock 'TRIGGER_ACCESS' is desired by: main
  20:04:16,008 DEBUG SimpleSemaphore:88 - Lock 'TRIGGER_ACCESS' is being obtained: main
  20:04:16,008 DEBUG SimpleSemaphore:105 - Lock 'TRIGGER_ACCESS' given to: main
        20:04:16,052 DEBUG SimpleSemaphore:132 - Lock 'TRIGGER_ACCESS' retuned by: main
        Found job: class to.test.cron.ImportProducts
        Tue Jul 15 20:05:00 CEST 2014
        isStarted=true
        isShutdown=false
        isInStandbyMode=false
        20:04:16,058 DEBUG QuartzSchedulerThread:276 - batch acquisition of 0 triggers
        20:04:42,961 ERROR ErrorLogger:2425 - An error occurred while scanning for the next triggers to fire.
        org.quartz.JobPersistenceException: Couldn't acquire next trigger: to.test.cron.ImportProducts [See nested exception: java.lang.ClassNotFoundException: to.test.cron.ImportProducts]
            at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2848)
            at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759)
            at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757)
            at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3787)
            at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756)
            at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
        Caused by: java.lang.ClassNotFoundException: to.test.cron.ImportProducts
            at java.net.URLClassLoader$1.run(Unknown Source)
            at java.net.URLClassLoader$1.run(Unknown Source)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at org.quartz.simpl.InitThreadContextClassLoadHelper.loadClass(InitThreadContextClassLoadHelper.java:72)
            at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:114)
            at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
            at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
            at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2816)
            ... 5 more

I don't quite understand why I am getting 3 completely different behaviours in the 3 differnt modes. Surley if the class can be found in RAM-mode, why should it not find it in jdbc mode? And why is it not being logged in clustered mode? The class is actually in a osgi-type module. Can that cause a problem (in jdbc-mode)? Is there anything I can do, so that the class can be found, like a passing the classloader etc to quartz?

I am pretty lost now and would really appreciate any help. It would be a shame to have to go back to standard cron jobs, especially as quartz has so much more to offer.

Thanks in advance for any help provided,

Michael

Answer 1

This is a general "problem" with a persistent job store. Your application apparently tries to add a job that already exists in the job store because it has already been added by your application in the past. You have two options:

1. You wipe out contents of your job store during the initialization of your application before you attempt to add jobs/triggers. Since Quartz 2.x, there is a new method Scheduler.clear() that you can use.

2. You modify your application code to deal with the fact that the job/trigger you are trying to add may be already present in the job store. If it is present, you simply update the job/trigger if necessary, or skip the job/trigger altogether.

When you think of it, this Quartz behavior actually makes sense, because jobs / triggers in the job store can be modified from outside of your application (e.g. by external systems using Quartz remote APIs).

You may also want to look into the XMLSchedulingDataProcessorPlugin that allows you to externalize job and trigger definitions from your application to an XML file/resource and it can deal with job/trigger name conflicts. This article provides an example of the XML file structure.

Hope this helps.