Metadata refresh deadlock (spring-security-saml)

Question

every couple of days our web app that uses Spring Security SAML has deadlock. Deadlock happens on refreshing metadata.

I tried too understand what is the problem from source code but without success.

This is stacktrace from three threads that are in deadlock:

1. Stack Trace Metadata-reload [136] (BLOCKED)

   org.opensaml.saml2.metadata.provider.AbstractMetadataProvider.initialize line: 402 
   org.springframework.security.saml.metadata.ExtendedMetadataDelegate.initialize line: 167 
   org.springframework.security.saml.metadata.MetadataManager.initializeProvider line: 398 
   org.springframework.security.saml.metadata.MetadataManager.refreshMetadata line: 246 
   org.springframework.security.saml.metadata.CachingMetadataManager.refreshMetadata line: 86 
   org.springframework.security.saml.metadata.MetadataManager$RefreshTask.run line: 1027 
   java.util.TimerThread.mainLoop line: 555 
   java.util.TimerThread.run line: 505

2. Stack Trace Timer-5 [135] (WAITING)

   sun.misc.Unsafe.park line: not available [native method]
   java.util.concurrent.locks.LockSupport.park line: 186 
   java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt line: 834 
   java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued line: 867 
   java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire line: 1197 
   java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock line: 945 
   org.springframework.security.saml.metadata.MetadataManager.setRefreshRequired line: 983 
   org.springframework.security.saml.metadata.MetadataManager$MetadataProviderObserver.onEvent line: 1047 
   org.opensaml.saml2.metadata.provider.ChainingMetadataProvider.emitChangeEvent line: 359 
   org.opensaml.saml2.metadata.provider.ChainingMetadataProvider$ContainedProviderObserver.onEvent line: 371 
   org.opensaml.saml2.metadata.provider.AbstractObservableMetadataProvider.emitChangeEvent line: 62 
   org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.processNonExpiredMetadata line: 427 
   org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.processNewMetadata line: 355 
   org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider.refresh line: 261 
   org.opensaml.saml2.metadata.provider.AbstractReloadingMetadataProvider$RefreshMetadataTask.run line: 513 
   java.util.TimerThread.mainLoop line: 555 
   java.util.TimerThread.run line: 505 

3. Stack Trace http-bio-7020-exec-548 [614] (WAITING)

   sun.misc.Unsafe.park line: not available [native method]
   java.util.concurrent.locks.LockSupport.park line: 186 
   java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt line: 834 
   java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared line: 964 
   java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared line: 1282 
   java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock line: 731 
   org.springframework.security.saml.metadata.CachingMetadataManager.getFromCacheOrUpdate line: 160 
   org.springframework.security.saml.metadata.CachingMetadataManager.getEntityDescriptor line: 116 
   org.springframework.security.saml.context.SAMLContextProviderImpl.populateLocalEntity line: 314 
   org.springframework.security.saml.context.SAMLContextProviderImpl.populateLocalContext line: 216 
   org.springframework.security.saml.context.SAMLContextProviderImpl.getLocalAndPeerEntity line: 126 
   org.springframework.security.saml.SAMLEntryPoint.commence line: 146 
   org.springframework.security.saml.SAMLEntryPoint.doFilter line: 107 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.web.FilterChainProxy.doFilterInternal line: 192 
   org.springframework.security.web.FilterChainProxy.doFilter line: 166 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter line: 199 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.web.authentication.logout.LogoutFilter.doFilter line: 110 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal line: 50 
   org.springframework.web.filter.OncePerRequestFilter.doFilter line: 106 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.web.session.ConcurrentSessionFilter.doFilter line: 125 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter line: 87 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.saml.metadata.MetadataGeneratorFilter.doFilter line: 87 
   org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter line: 342 
   org.springframework.security.web.FilterChainProxy.doFilterInternal line: 192 
   org.springframework.security.web.FilterChainProxy.doFilter line: 160 
   org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate line: 343 
   org.springframework.web.filter.DelegatingFilterProxy.doFilter line: 260 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 241 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 208 
   org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal line: 88 
   org.springframework.web.filter.OncePerRequestFilter.doFilter line: 106 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 241 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 208 
   hr.isvu.studomat.web.filter.RequestLoggerFilter.proslijediObraduZahtjeva line: 126 
   hr.isvu.studomat.web.filter.RequestLoggerFilter.doFilter line: 57 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 241 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 208 
   org.apache.catalina.core.StandardWrapperValve.invoke line: 220 
   org.apache.catalina.core.StandardContextValve.invoke line: 122 
   org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 501 
   org.apache.catalina.core.StandardHostValve.invoke line: 171 
   org.apache.catalina.valves.ErrorReportValve.invoke line: 102 
   org.apache.catalina.valves.AccessLogValve.invoke line: 950 
   org.apache.catalina.core.StandardEngineValve.invoke line: 116 
   org.apache.catalina.connector.CoyoteAdapter.service line: 408 
   org.apache.coyote.http11.AbstractHttp11Processor.process line: 1040 
   org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process line: 607 
   org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run line: 314 
   java.util.concurrent.ThreadPoolExecutor.runWorker line: 1145 
   java.util.concurrent.ThreadPoolExecutor$Worker.run line: 615 
   org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
   java.lang.Thread.run line: 722 

We use:

1. spring-security-saml2-core 1.0.0.RELEASE
2. org.opensaml.opensaml 2.6.1

This is metadata refresh config:

...
        <!-- IDP Metadata configuration - paths to metadata of IDPs in circle of
                trust is here -->
        <bean id="metadata"
                class="org.springframework.security.saml.metadata.CachingMetadataManager">
                <constructor-arg>
                        <list>
                                <bean class="org.opensaml.saml2.metadata.provider.HTTPMetadataProvider">
                                        <constructor-arg>
                                                <value>https://www.example.org/saml2/idp/metadata.php</value>
                                        </constructor-arg>
                                        <constructor-arg>
                                                <value type="int">5000</value>
                                        </constructor-arg>
                                        <property name="parserPool" ref="parserPool" />
                                </bean>
                        </list>
                </constructor-arg>
        </bean>
...

How can we solve this deadlock?

Thanks in advance, Denis

Answer 1

This is a valid issue, I opened a ticket in Jira and pushed a fix to master. There should be a new build available tomorrow at snapshot repo, can you please re-test with it?