Spring Scheduled Jobs executing 3 times in different pools

Question

I have a Spring @Scheduled job that runs hourly, but I see that it's actually running 3 times each hour. Here's the log output that shows this problem:

2013-05-06 12:00:27,656 [pool-2-thread-1] INFO  src.jobs.NotifyUsersWhenVideoAvailableJob - Emails sent from NotifyUsersWhenVideoAvailableJob: 1
2013-05-06 12:00:27,750 [pool-1-thread-1] INFO  src.jobs.NotifyUsersWhenVideoAvailableJob - Emails sent from NotifyUsersWhenVideoAvailableJob: 1
2013-05-06 12:00:27,796 [pool-4-thread-1] INFO  src.jobs.NotifyUsersWhenVideoAvailableJob - Emails sent from NotifyUsersWhenVideoAvailableJob: 1

This is obviously really annoying, as three copies of the same email are going out each time this job runs.

I'm using Spring 3.1

Here's my configuration:

WEB.XML

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
  version="2.5">
  <display-name>site2</display-name>
  <description>Roo generated site2 application</description>
  <context-param>
    <param-name>defaultHtmlEscape</param-name>
    <param-value>true</param-value>
  </context-param>
  <context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>classpath*:META-INF/spring/applicationContext*.xml</param-value>
  </context-param>
  <filter>
    <filter-name>springSecurityFilterChain</filter-name>
    <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
  </filter>

  <filter-mapping>
    <filter-name>springSecurityFilterChain</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
  <filter>
    <filter-name>CharacterEncodingFilter</filter-name>
    <filter-class>org.springframework.web.filter.CharacterEncodingFilter</filter-class>
    <init-param>
      <param-name>encoding</param-name>
      <param-value>UTF-8</param-value>
    </init-param>
    <init-param>
      <param-name>forceEncoding</param-name>
      <param-value>true</param-value>
    </init-param>
  </filter>
  <filter>
    <filter-name>HttpMethodFilter</filter-name>
    <filter-class>org.springframework.web.filter.HiddenHttpMethodFilter</filter-class>
  </filter>
  <filter>
    <filter-name>Spring OpenEntityManagerInViewFilter</filter-name>
    <filter-class>org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter</filter-class>
  </filter>
  <filter-mapping>
    <filter-name>CharacterEncodingFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
  <filter-mapping>
    <filter-name>HttpMethodFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
  <filter-mapping>
    <filter-name>Spring OpenEntityManagerInViewFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
  <listener>
    <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
  </listener>
  <servlet>
    <servlet-name>site2</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
    <init-param>
      <param-name>contextConfigLocation</param-name>
      <param-value>WEB-INF/spring/webmvc-config.xml</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>site2</servlet-name>
    <url-pattern>/</url-pattern>
  </servlet-mapping>
  <session-config>
    <session-timeout>120</session-timeout>
  </session-config>
  <error-page>
    <exception-type>java.lang.Exception</exception-type>
    <location>/error</location>
  </error-page>
</web-app>

ApplicationContext.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:aop="http://www.springframework.org/schema/aop" xmlns:context="http://www.springframework.org/schema/context"
  xmlns:jee="http://www.springframework.org/schema/jee" xmlns:tx="http://www.springframework.org/schema/tx"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:security="http://www.springframework.org/schema/security"
  xmlns:task="http://www.springframework.org/schema/task"
  xsi:schemaLocation="http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.1.xsd
           http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
           http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd
           http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-3.1.xsd
           http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.1.xsd
           http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-3.1.xsd
           http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd">

  <context:property-placeholder
    location="classpath*:META-INF/spring/*.properties" />

  <context:spring-configured />

  <context:component-scan base-package="src">
    <context:exclude-filter expression=".*_Roo_.*"
      type="regex" />
    <context:exclude-filter expression="org.springframework.stereotype.Controller"
      type="annotation" />
  </context:component-scan>

  <task:annotation-driven/>

  <bean id="sessionFactory"
    class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
    <property name="dataSource" ref="dataSource" />
    <property name="packagesToScan" value="src.domain" />
    <property name="mappingDirectoryLocations">
      <list>
        <value>classpath*:**/src.domain</value>
      </list>
    </property>
    <property name="hibernateProperties">
      <props>
        <prop key="hibernate.dialect">org.hibernate.dialect.MySQL5Dialect</prop>
        <prop key="hibernate.show_sql">true</prop>
        <prop key="format_sql">true</prop>
        <prop key="hibernate.use_sql_comments">true</prop>
      </props>
    </property>
  </bean>

  <bean id="webexpressionHandler" class="org.springframework.security.web.access.expression.DefaultWebSecurityExpressionHandler" />
  <security:http pattern="/index.html" security="none" />
  <security:http pattern="/about.html" security="none" />
  <security:http pattern="/pricing.html" security="none" />
  <security:http pattern="/signup.html" security="none" />
  <security:http pattern="/forgotPassword.htm" security="none" />
  <security:http pattern="/**.json" security="none" />

  <security:http auto-config="true">
    <security:intercept-url pattern="/**.htm"
      access="ROLE_FREE" />
      <security:intercept-url pattern="/test/**.htm"
      access="ROLE_FREE" />
      <security:intercept-url pattern="/admin.htm"
      access="ROLE_SUPERUSER" />
      <security:intercept-url pattern="/exerciseFiles/**.zip"
      access="ROLE_RECOMMENDED" />
    <security:form-login login-page="/login.html"
      authentication-failure-handler-ref="failedLoginService"
      authentication-success-handler-ref="successfulLoginService" />
      <security:logout logout-success-url="/index.html"/>
  </security:http>

  <security:authentication-manager>
    <security:authentication-provider
      user-service-ref="userDetailsService" />
  </security:authentication-manager>
</beans>

webmvc-config.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:context="http://www.springframework.org/schema/context" xmlns:mvc="http://www.springframework.org/schema/mvc" xmlns:p="http://www.springframework.org/schema/p" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd                 http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd                 http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.1.xsd">

    <context:component-scan base-package="src" use-default-filters="false">
        <context:include-filter expression="org.springframework.stereotype.Controller" type="annotation"/>
    </context:component-scan>

    <mvc:annotation-driven conversion-service="applicationConversionService"/>

    <mvc:resources location="/, classpath:/META-INF/web-resources/" mapping="/resources/**"/>

    <mvc:default-servlet-handler/>

    <mvc:interceptors>
        <bean class="org.springframework.web.servlet.theme.ThemeChangeInterceptor"/>
        <bean class="org.springframework.web.servlet.i18n.LocaleChangeInterceptor" p:paramName="lang"/>
    </mvc:interceptors>

    <mvc:view-controller path="/" view-name="index"/>
    <mvc:view-controller path="/uncaughtException"/>
    <mvc:view-controller path="/resourceNotFound"/>
    <mvc:view-controller path="/dataAccessFailure"/>

    <bean class="org.springframework.context.support.ReloadableResourceBundleMessageSource" id="messageSource" p:basenames="WEB-INF/i18n/messages,WEB-INF/i18n/application" p:fallbackToSystemLocale="false"/>

    <bean class="org.springframework.web.servlet.i18n.CookieLocaleResolver" id="localeResolver" p:cookieName="locale"/>

    <bean class="org.springframework.ui.context.support.ResourceBundleThemeSource" id="themeSource"/>

    <bean class="org.springframework.web.servlet.theme.CookieThemeResolver" id="themeResolver" p:cookieName="theme" p:defaultThemeName="standard"/>

    <bean class="org.springframework.web.servlet.handler.SimpleMappingExceptionResolver" p:defaultErrorView="uncaughtException">
        <property name="exceptionMappings">
            <props>
                <prop key=".DataAccessException">dataAccessFailure</prop>
                <prop key=".NoSuchRequestHandlingMethodException">resourceNotFound</prop>
                <prop key=".TypeMismatchException">resourceNotFound</prop>
                <prop key=".MissingServletRequestParameterException">resourceNotFound</prop>
            </props>
        </property>
    </bean>

    <bean class="org.springframework.web.multipart.commons.CommonsMultipartResolver" id="multipartResolver"/>

    <bean class="org.springframework.web.servlet.view.UrlBasedViewResolver" id="viewResolver">
        <property name="viewClass" value="org.springframework.web.servlet.view.JstlView"/>
        <property name="prefix" value="/WEB-INF/jsp/"/>
        <property name="suffix" value=".jsp"/>
    </bean>

    <bean class="src.web.ApplicationConversionServiceFactoryBean" id="applicationConversionService"/>

</beans>

And here's the Class file where the jobs are being executed:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;

import src.jobs.NotifyUsersWhenVideoAvailableJob;
import src.jobs.PayAsYouGoReminderJob;
import src.jobs.RemindUsersToActivateJob;

@Service
public class ScheduledJobsService
{
  @Autowired
  @Qualifier("videoJob")
  private NotifyUsersWhenVideoAvailableJob videoJob;
  @Autowired
  @Qualifier("activateJob")
  private RemindUsersToActivateJob activateJob; 
  @Autowired
  private PayAsYouGoReminderJob payAsYouGoReminderJob;

  //This cron just should be set to 1 second past the hour
  // as the videoJob has dates set to be ON the hour exactly
  // example of good setting: @Scheduled(cron="1 0 * * * *")
  @Scheduled(cron="1 0 * * * *")
  public void doHourlyJobs() 
  {
    videoJob.run();
  }

  @Scheduled(cron="0 0 12 * * *")
  public void doDailyJobs() 
  {
    try
    {
        activateJob.run();
    }
    catch (Exception e)
    {
      EmailService.sendError(e, null);
    }

    try
    {
      payAsYouGoReminderJob.run();
    }
    catch (Exception e)
    {
      EmailService.sendError(e, null);
    }
  }
}

* EDIT *

After doing some more poking around I've narrowed in (a little more) on where the problem is likely happening. I cannot reproduce this problem in my DEV environment, so there must be some sort of configuration on my PROD box.

My PROD box has 5 different web applications in the webapps folder:

• tomcat 6.0
  • webapps
    • site1
    • site2
    • site3
    • site4
    • site5

I made some changes to my server.xml file and now it only seems to be executing the jobs twice instead of three times. Here's the new configuration:

server.xml

<Server port="8005" shutdown="SHUTDOWN">

  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <Service name="Catalina">
    <Connector port="80" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>
      <Host name="site1.net"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
        <Alias>www.site1.net</Alias>
      </Host>

        <Host name="site2.net"  appBase="webapps"
            unpackWARs="true" autoDeploy="false" deployOnStartup="false"
            xmlValidation="false" xmlNamespaceAware="false">
            <Alias>www.site2.net</Alias>
            <Context path="" docBase="./site2"/>
        </Host>

    </Engine>
  </Service>
</Server>

Answer 1

If you define two web applications in your server.xml, and you're not careful (see below), then you'll have two completely separate instances of everything running in the servlet container.

The context.xml file gives you a way to differentiate between things that are shared and things that are application-specific. The most common example of a thing that can be shared is a datasource or connection pool. This doesn't really help you with Spring beans though.

Fortunately Spring gives you a way to define a shared parent application context, so that beans are shared across WARs running in the same container. It's pretty well documented online.

Unfortunately I don't think this works across <Host> elements. Virtual hosts allow you to isolate resources on the same physical machine, specifically so that the resources can be managed independently. So, if both of the applications define the tasks via <task:annotation-driven/>, then you end up with two separate instances.

So, in sum, if you want a single instances of each task per servlet container, then you need a shared application context. Can you:

• Move the applications into one <Host>
• Use the <Alias> tag with the single <Host> to support two separate URLs
• Use the above mentioned techniques to make sure that there is a shared application context