MongoCleaner threads accumulating on Spring Data app

Question

I have a java website using Spring Data to provide integration with MongoDB. The application runs well, but keeps accumulating threads, until the server eventually goes down with this error:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mongo': Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
2013-10-15T14:51:40.305986+00:00 app[web.1]: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mongo': Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
2013-10-15T14:51:40.305986+00:00 app[web.1]: 1343234 [http-nio-55996-exec-6] DEBUG org.springframework.web.servlet.DispatcherServlet  - Could not complete request
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1455)
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at org.springframework.contex
t.support.GenericXmlApplicationContext.<init>(GenericXmlApplicationContext.java:71)
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
2013-10-15T14:51:40.305986+00:00 app[web.1]:    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:591)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:469)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at com.jux.model.Signin.persistSignin(Signin.java:40)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at com.intuit.controller.LoginController.authenticateUser(LoginController.java:71)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at java.lang.reflect.Method.invoke(Method.java:616)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.springframework.web.servlet.FrameworkServlet.
processRequest(FrameworkServlet.java:882)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2013-10-15T14:51:40.306169+00:00 app[web.1]:    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
2013-10-15T14:51:40.306727+00:00 app[web.1]:    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
2013-10-15T14:51:40.306727+00:00 app[web.1]:    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
2013-10-15T14:51:40.306727+00:00 app[web.1]:    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
2013-10-15T14:51:40.306334+00:00 app[web.1]:    at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
2013-10-15T14:51:40.306727+00:00 app[web.1]:    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
2013-10-15T14:51:40.306875+00:00 app[web.1]:    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
2013-10-15T14:51:40.306875+00:00 app[web.1]: Caused by: java.lang.OutOfMemoryError: unable to create new native thread
2013-10-15T14:51:40.306875+00:00 app[web.1]:    at java.lang.Thread.start0(Native Method)
2013-10-15T14:51:40.306727+00:00 app[web.1]:    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
2013-10-15T14:51:40.306727+00:00 app[web.1]:    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
2013-10-15T14:51:40.306875+00:00 app[web.1]:    at com.mongodb.Mongo.<init>(Mongo.java:192)
2013-10-15T14:51:40.306875+00:00 app[web.1]:    at java.lang.Thread.start(Thread.java:657)

I have tried to update the version of Spring Data I am using, and tried with both 1.3.1.RELEASE and 1.3.2.BUILD-SNAPSHOT - same issue.

I have also read that the Mongo driver is supposed to close these threads, and nothing needs to be done on my end to close threads, but I'm starting to doubt. Any idea how to solve this issue?

Here is my mongodb.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xmlns:context="http://www.springframework.org/schema/context"
          xmlns:mongo="http://www.springframework.org/schema/data/mongo"
          xsi:schemaLocation=
          "http://www.springframework.org/schema/context
          http://www.springframework.org/schema/context/spring-context-3.0.xsd
          http://www.springframework.org/schema/data/mongo
          http://www.springframework.org/schema/data/mongo/spring-mongo-1.0.xsd
          http://www.springframework.org/schema/beans
          http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

    <!-- Default bean name is 'mongo' -->
    <mongo:mongo host="xxxxxx.mongolab.com" port="12345">
        <mongo:options connections-per-host="8"
                   threads-allowed-to-block-for-connection-multiplier="4"
                   connect-timeout="1000"
                   max-wait-time="1500"
                   auto-connect-retry="true"
                   socket-keep-alive="true"
                   socket-timeout="1500"
                   slave-ok="true"
                   write-number="1"
                   write-timeout="0"
                   write-fsync="true"/>
    </mongo:mongo>
    <mongo:db-factory id="myMongoDbFactory"
                  host="xxxxx.mongolab.com"
                  port="12345"
                  dbname="zzzz"
                  username="yyyy"
                  password="xxxx"/>

    <bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
        <constructor-arg ref="myMongoDbFactory"/>
    </bean>

</beans>

This is how I make queries to MongoDB in the controllers:

@RequestMapping(value = "/deletePart.htm", method = RequestMethod.GET)
public String deletePart(String part_id) {
    LOG.debug("in deletePart, part_id: "+part_id);
    ApplicationContext ctx = new GenericXmlApplicationContext("mongodb.xml");
    MongoOperations mongoOps = (MongoOperations)ctx.getBean("mongoTemplate");
    Part p = mongoOps.findOne(new Query(where("id").is(part_id)), Part.class);
    mongoOps.remove(p);     
    return "redirect:/parts.htm";
}

Here is a picture of how threads keep accumulating and never closing:

and from the thread dump, here is the detail of one hung thread:

"MongoCleaner1448498515" daemon prio=5 tid=0x00007f939f888000 nid=0x12f03 waiting on condition [0x00000001ac920000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.mongodb.Mongo$CursorCleanerThread.run(Mongo.java:770)

   Locked ownable synchronizers:
    - None

Answer 1

If you close the application context, that will close the Mongo instance and prevent the unrestrained thread growth:

@RequestMapping(value = "/deletePart.htm", method = RequestMethod.GET)
public String deletePart(String part_id) {
   LOG.debug("in deletePart, part_id: "+part_id);
   GenericXmlApplicationContext ctx = new GenericXmlApplicationContext("mongodb.xml");
   try {
      MongoOperations mongoOps = (MongoOperations)ctx.getBean("mongoTemplate");
      Part p = mongoOps.findOne(new Query(where("id").is(part_id)), Part.class);
      mongoOps.remove(p);     
      return "redirect:/parts.htm";
   } finally {
      ctx.close();   // close the context, will close the Mongo instance
   }
}

But even though this will work, your application still should not create an application context instance per request.