How to exit SQS read in Java SDK

Question

We have a Java web service which polls an AWS SQS queue for messages. It uses the maximum wait time of 20 seconds. When the service is updated in Tomcat, Tomcat attempts to shutdown the existing service and then starts the updated version. When this happens, if the queue is being read, the thread is stuck and Tomcat is unable to stop it, resulting in a memory leak. From my experimenting this memory leak only happens if Tomcat tries to stop the service while it is waiting on the queue.

How can I stop an SQS queue in Java during the shutdown process?

Here are some more details.

Jan 03, 2014 7:52:52 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application appears to have started a thread named [pollSQSQueue-1] but has failed to stop it. This is very likely to create a memory leak.
Jan 03, 2014 7:52:52 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application created a ThreadLocal with key of type [com.amazonaws.auth.AWS4Signer$2] (value [com.amazonaws.auth.AWS4Signer$2@71f69c90]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@ef87e460]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 03, 2014 7:52:52 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application created a ThreadLocal with key of type [com.amazonaws.auth.AWS4Signer$1] (value [com.amazonaws.auth.AWS4Signer$1@75a0ec63]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@6f64295a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

The service uses Spring and uses a Spring scheduler to start the polling process.

<task:scheduler id="pollSQSQueue" pool-size="1"/>

I have tried creating a Spring ApplicationListener<ContextClosedEvent> to try to shutdown the queue. My first attempt was to try to shutdown the queue and see if it handled it gracefully. It did not.

@Override
public void onApplicationEvent(ContextClosedEvent contextClosedEvent)
{
    amazonSQS.shutdown();
}

The next thing I tried was to use the AmazonSQSAsync and use an async retrieve. I then obtained the Future<ReceiveMessageResult> and tried to call cancel on it during the Spring shutdown. This was also not handled very gracefully.

---

Also, I am aware of the need to shutdown the IdleConnectionReaper and do this when Spring is shutting down, so I don't think it is related to that.

Answer 1

You can create a separate thread for the polling work. Something along these lines:

class MyService extends/implements ... {
    private SQSPollingThread pollingThread;

    //this is just my guess of what the "service start" method looks like
    @Override
    public void onApplicationEvent(ContextStartedEvent contextStartedEvent)
    {
         pollingThread = new SQSPollingThread();
         pollingThread.start();
    }

    @Override
    public void onApplicationEvent(ContextClosedEvent contextClosedEvent)
    {
        pollingThread.stop();
    }
}

class SQSPollingThread implements Runnable() {
    private volatile boolean stopped = false;

    public void stop() {
        stopped = true;
    }

    @Override
    public void run() {
        while (!stopped) {
            message = pollSqsBlockingFor20seconds();
            process(message);
        }
    }
}

With this setup, when TomCat "tells" your service to stop, your service will set the stopped flag and return immediately. There may still be a polling request pending, which will ultimately finish within the next 20 seconds and get processed. After this last event is processed, the inner SQSPollingThread will also end.