Spring MVC WebApp : @schedule: java-sdk-http-connection-reaper : Failed to Stop

Question

I have a Web Application(using Spring 3.1) which uses @Scheduled Annotation for periodically executing a worker task (scheduled delay). The worker task opens up a connection to AWS DynamoDb and does some DB read/updates. When I stop the webapp (from Tomcat manager) I get this message in catalina.out:

"SEVERE: The web application [] appears to have started a thread named [java-sdk-http-connection-reaper] but has failed to stop it. This is very likely to create a memory leak."

I get a feeling that this has something to do with my scheduled task still running even after Tomcat stops.

@Service
public class TaskScheduler implements ApplicationListener<ContextClosedEvent>{

@Autowired
private WorkerTask workerTask;

AmazonDynamoDBClient myDbConn = null;

   private TaskScheduler() {    
   myDbConn = new AWSConnector("aws.properties").getDynamoConnection();
   }

/*
 * Will be repeatedly called, 10 seconds after the finish of the previous 
 * invocation.
 */
@Scheduled(fixedDelay=100000)
public void process() {
    System.out.println("Scheduling worker task");
            //worker task does some db read/writes
    Future<String> status = workerTask.work(myDbConn);
    if (status.isDone()) {
        System.out.println("Completed Task");
        return;
    }

}

@Override
public void onApplicationEvent(ContextClosedEvent arg0) {
    if(event instanceof ContextClosedEvent) {   
      // TODO Auto-generated method stub
      if(myDbConn != null) {
        this.myDbConn.shutdown();
      }
          }

}

dispatcher-servlet.xml:

<task:annotation-driven scheduler="taskScheduler"/>
<task:scheduler id="taskScheduler" pool-size="2"/>
......
<bean id="TaskScheduler" class="com.sample.TaskScheduler"/>

Am I doing this correctly? a) I don't explicitly start the TaskScheduler. So i'm assuming spring takes care of starting this service. The 'this.myDbConn.shutdown()' is called. Despite this, I get the error. I'm using Spring MVC.

Answer 1

This is likely caused by the AWS library which starts a thread in the background called com.amazonaws.http.IdleConnectionReaper

You can shut it down by implementing a ServletContextListener to close it on shutdown

public class YourListener implements ServletContextListener {

@Override
public void contextInitialized(ServletContextEvent contextEvent) {

}

@Override
public void contextDestroyed(ServletContextEvent contextEvent) {

    try {
        com.amazonaws.http.IdleConnectionReaper.shutdown();
    } catch (Throwable t) {
        // log the error
    }
}
}

and adding this to your web.xml

<listener>
    <listener-class>
        your.package.YourListener 
    </listener-class>
</listener>

Answer 2

I had this issue as well but I decided on an alternative solution to @David_Wartell's above.

I tracked down the class that was creating an offending object/objects from Amazon's aws-java-sdk library that were starting up the IdleConnectionReaper thread but never shutting down (these were com.amazonaws.services.ec2.AmazonEC2Client and com.amazonaws.services.cloudwatch.AmazonCloudWatchClient). Then I added a destroy() method to this class which called the static method com.amazonaws.http.IdleConnectionReaper.shutdown(). The destroy method is called when the class is garbage collected and it is configured using Spring applicationContext.xml. The advantages of doing it this way are that it will work even for non web applications and it de-couples the thread shutdown from your web context. The proper solution is that the classes from the amazon aws-java-sdk library that start the IdleConnectionReaper thread should shut it down but they don't - hence this bug. See references and code snippets below for my solution:

applicationContext.xml

<bean id="YourBeanName" class="com.your.package.name.YourBeanName" destroy-method="destroy">
    <!-- other optional configuration goes here -->
</bean>

YourBeanName.java - (class that creates the offending amazon objects)

    public class YourBeanName {

        // omitted code

        public void destroy() {
            com.amazonaws.http.IdleConnectionReaper.shutdown();
        }

        // omitted code

    }

references:

Amazon forum - Shutting down IdleConnectionReaper
Spring docs - Customizing the nature of a bean

Answer 3

On top of Stuart's answer (and assuming you are using Spring), an alternative if you don't use XML configuration files:

@Component
public class MyBean {

// ...

    @PreDestroy
    private void cleanUp() {
        try {
            // Shutting down AWS IdleConnectionReaper thread...
            com.amazonaws.http.IdleConnectionReaper.shutdown();
        } catch (Throwable t) {
            // log error
        }
    }

}

It has worked for me, when I used a bean implementing com.amazonaws.services.s3.AmazonS3 interface.