Spring PermGen space issue

Question

I've been having a pretty serious production issue and I was hoping that someone could help out.

I am using Spring 3.1.0 and have been impressed by its capabilities.

During development, everything was fine. Unfortunately, upon deploying to production, we came up against some issues.

We are getting PermGen space out of memory errors.

On our dev/test box, we have the following settings: -Xms2048m -Xmx2048m -XX:PermSize=512m -XX:MaxPermSize=1024m (please note that we are not confident that these are correct settings)

On our production box, we have the following settings: -Xms3056m -Xmx3056m -XX:PermSize=128m -XX:MaxPermSize=512m (again, please note that we are not confident that these are correct settings)

Admittedly, the production environment is quite different in that it has a lot more applications running on it.

According to what I've read here, we should set the max and min to the same value and the maxpermsize to 1/4 of either one of those.

I have read in a number of locations (including here) that it may be unavoidable to restart a server due to leaks. Does anybody know if this is true?

Also, does anybody know how much memory would typically be needed by the Spring framework? Upon deployment of the application, I find that the amount of memory used increases by approximately 345 MB. I understand that a number of classes need to be instantiated on startup of the application but this seems like a lot. Do I have something setup incorrectly? We are only using Spring JDBC at the moment. Is there some way to limit the framework to those classes only?

Our code base is rather large and I understand that JAXB can cause serious problems.

I have seen recommendations of using

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

here and here. Are there any possible negative side effects to using these arguments?

So, to summarise:

1. Are the settings we have correct and, if not, what should they be?

2. Are server restarts from time to time unavoidable with the Spring framework?

3. How much memory should Spring be taking up? Is there a way to limit the classes used?

4. Are there any possible side effects to using -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled ?

Any and all help would be greatly appreciated. Thanks in advance.

The stack trace is below (there is more but I couldn't post it all):

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter]: Constructor threw exception; nested exception is java.lang.OutOfMemoryError: PermGen space
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:997)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:943)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:286)
    at org.springframework.web.servlet.DispatcherServlet.createDefaultStrategy(DispatcherServlet.java:788)
    at org.springframework.web.servlet.DispatcherServlet.getDefaultStrategies(DispatcherServlet.java:757)
    at org.springframework.web.servlet.DispatcherServlet.initHandlerAdapters(DispatcherServlet.java:565)
    at org.springframework.web.servlet.DispatcherServlet.initStrategies(DispatcherServlet.java:422)
    at org.springframework.web.servlet.DispatcherServlet.onRefresh(DispatcherServlet.java:410)
    at org.springframework.web.servlet.FrameworkServlet.onApplicationEvent(FrameworkServlet.java:752)
    at org.springframework.web.servlet.FrameworkServlet$ContextRefreshListener.onApplicationEvent(FrameworkServlet.java:989)
    at org.springframework.web.servlet.FrameworkServlet$ContextRefreshListener.onApplicationEvent(FrameworkServlet.java:1)
    at org.springframework.context.event.GenericApplicationListenerAdapter.onApplicationEvent(GenericApplicationListenerAdapter.java:51)
    at org.springframework.context.event.SourceFilteringListener.onApplicationEventInternal(SourceFilteringListener.java:97)
    at org.springframework.context.event.SourceFilteringListener.onApplicationEvent(SourceFilteringListener.java:68)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:324)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:929)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
    at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:631)
    at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:588)
    at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:645)
    at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:508)
    at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:449)
    at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:133)
    at javax.servlet.GenericServlet.init(GenericServlet.java:241)
    at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
    at weblogic.servlet.internal.StubSecurityHelper.createServlet(StubSecurityHelper.java:64)
    at weblogic.servlet.internal.StubLifecycleHelper.createOneInstance(StubLifecycleHelper.java:58)
    at weblogic.servlet.internal.StubLifecycleHelper.<init>(StubLifecycleHelper.java:48)
    at weblogic.servlet.internal.ServletStubImpl.prepareServlet(ServletStubImpl.java:539)
    at weblogic.servlet.internal.WebAppServletContext.preloadServlet(WebAppServletContext.java:1976)
    at weblogic.servlet.internal.WebAppServletContext.loadServletsOnStartup(WebAppServletContext.java:1950)
    at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1869)
    at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3126)
    at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1512)
    at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:486)
    at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
    at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
    at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
    at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedModuleDriver.java:200)
    at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:247)
    at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
    at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
    at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
    at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:27)
    at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:1267)
    at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
    at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:409)
    at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:58)
    at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
    at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
    at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:569)
    at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:150)
    at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:116)
    at weblogic.deploy.internal.targetserver.operations.StartOperation.doCommit(StartOperation.java:143)
    at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:323)
    at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:844)
    at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1253)
    at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:440)

Answer 1

In order to see the permGen VM, you can look at MemoryPool MBean in JConsole. Also, you can add the following JVM arguments (in case you don't have already)

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/msp/log 

for heapdumps collection.

-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/opt/msp/log/gc.log` 

for GC related information.

This will help in understanding Garbage Collection pattern and may guide you in coming up with a better limits.