Weblogic and Oracle: Stuck ExecuteThread

Question

We are running SpringMVC with Hibernate on a clustered Weblogic environment. Our application worked well, until we run into database space problems. Due to that, our DB provider extended the database storage.

After that change the Weblogic system status is complaining about the thread pool "Thread pool has stucked threads".

The stack-trace in the log file looks as follows:

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.ke
rnel.Default (self-tuning)' has been busy for "641" seconds working on the request "weblogic.servlet.internal.ServletReq
uestImpl@17782ff3[
GET /default/default/applicationPreloader/load.do?_=1331887527968 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2
Accept: */*
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
x-requested-with: XMLHttpRequest
Referer: http://novintegweb01.eu.acme.net:9136/default/default/applicationPreloader.do
Cookie: JSESSIONID1=NhBdPv9Hx8Wj5lnr5vnGhLSbR2GGM1DPhqT2YGzWCGFvhN3WJWVH!700169023!1391220923
Connection: Keep-Alive
Proxy-Client-IP: 192.168.1.1
X-Forwarded-For: 192.168.1.1
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: G4kGT1kfuDBbtj5w//SYNt6hlVo

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
        java.net.SocketInputStream.socketRead0(Native Method)
        java.net.SocketInputStream.read(SocketInputStream.java:129)
        oracle.net.ns.Packet.receive(Packet.java:293)
        oracle.net.ns.DataPacket.receive(DataPacket.java:104)
        oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
        oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
        oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
        oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
        oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
        oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
        oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)
        oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)
        oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
        oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
        oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
        oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
        oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1009)
        oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:10117)
        oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10221)
        oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:216)
        weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:210)
        org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
        org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:247)
        org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:93)
        org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:88)
        org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:223)
        org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1119)
        org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:39)
        org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
        org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
        org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:171)
        org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
        org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
        org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1001)
        org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:339)
        org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
        org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
        org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
        org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
        org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
        org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        com.acme.superproject.dao.CertificationDaoImpl$$EnhancerByCGLIB$$bf3135a5.getById(<generated>)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDateFromWs(UserProfileSabaServiceImpl.java:436)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDate(UserProfileSabaServiceImpl.java:401)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.createCertificationRelationObject(UserProfileSabaServiceImpl.java:373)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.getRequiredLearning(UserProfileSabaServiceImpl.java:334)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.extractUserProfile(UserProfileSabaServiceImpl.java:203)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:73)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:114)
        com.acme.superproject.integration.UserProfileSabaServiceImpl$$FastClassByCGLIB$$e48c61c9.invoke(<generated>)
        net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
        sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        java.lang.reflect.Method.invoke(Method.java:597)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        com.acme.superproject.integration.UserProfileSabaServiceImpl$$EnhancerByCGLIB$$5d7b85df.getUserProfile(<generated>)
        com.acme.superproject.service.UserProfileServiceImpl.getUserProfile(UserProfileServiceImpl.java:26)
        com.acme.superproject.service.UserProfileServiceImpl$$FastClassByCGLIB$$70cfe815.invoke(<generated>)
        net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
        sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        java.lang.reflect.Method.invoke(Method.java:597)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        com.acme.superproject.service.UserProfileServiceImpl$$EnhancerByCGLIB$$5bcb212b.getUserProfile(<generated>)
        com.acme.superproject.web.session.SessionDataProvider.fetchUserProfile(SessionDataProvider.java:97)
        com.acme.superproject.web.session.SessionDataProvider.getUserProfile(SessionDataProvider.java:78)
        com.acme.superproject.web.session.SessionDataProvider.fetchUserRegistrationCollection(SessionDataProvider.java:155)
        com.acme.superproject.web.session.SessionDataProvider.getUserRegistrationCollection(SessionDataProvider.java:141)
        com.acme.superproject.web.ApplicationPreloaderController.loadAction(ApplicationPreloaderController.java:84)
        sun.reflect.GeneratedMethodAccessor6506.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        java.lang.reflect.Method.invoke(Method.java:597)
        org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
        org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426)
        org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414)
        org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
        org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
        org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
        org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
        weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
        weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
        weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
        weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368)
        org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
        org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169)
        org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
        weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
        weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
        weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
        weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
        weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
        weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
        weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
>

We already have restarted Weblogic, the database server and made a new deployment. The problem is always the same: When we try to access the login page of the application it hangs up - as far as I can see while trying to perform a DB transaction.

In addition to that, when I try to connect to our database with "Oracle SQL Developer" we also get the error "ORA-00257: archiver error. Connect internal only, until freed." Unfortunately we have no SYSDBA rights (because managed by an external provider).

My question are:

• Has anyone of you had a similar problem before?
• Do you think the root cause of such problems is more on application side, or more on DB side?
• How can I at least get rid of the stucked threads?

Answer 1

Chances are that the database storage is full, or that the archiver cannot move or create new archive logs. This is almost definitely an issue at the database level and not the app server level. Since you are getting this message that means you already have a connection to the database, so it is not sitting and waiting for a connection.

Stuck Threads in WebLogic don't always need to be 'fixed' and are not always a problem. The purpose of the stuck thread mechanism is to notify you when a thread is taking an abnormally long time to do something (like access the database). The stuck thread notification is just that - a notification that something might not be right, and in this case it is serving that purpose.

If you have threads that routinely take 20 minutes to do something but want to be notified if it takes longer than 25 minutes, then have to increase the Stuck Thread Limit to 25 minutes and get the notification at that point then you must put this limit from 600 seconds to 1500 seconds (25*60).

You can edit stuck thread limit in weblogic admin base_domain > Environment > Servers > Admin Server > Configuration/Tuning.

refer below link : https://docs.oracle.com/cd/E19316-01/820-3745/ghudn/index.html