LDAP requests hanging for 15 seconds

Question

I have JBoss application server that is using LDAP for authentication. Lately we have noticed that there are a lot of slow requests (> 15 seconds).

I did some threaddumps of the server and noticed that many threads where waiting on a lock: com.sun.jndi.ldap.LdapRequest@54ceac

java.lang.Object.wait(Native Method)
com.sun.jndi.ldap.Connection.readReply(Connection.java:418)
com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:340)
com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:192)
com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2637)
com.sun.jndi.ldap.LdapCtx.(LdapCtx.java:283)
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:134)
com.sun.jndi.url.ldap.ldapURLContextFactory.getObjectInstance(ldapURLContextFactory.java:35)
javax.naming.spi.NamingManager.getURLObject(NamingManager.java:584)

All of the requests I have seen that have been waiting in this state have used more than 15 seconds to complete. We are monitoring the LDAP-server and all requests from the monitoring tool finish in less than 200 ms. This makes me think this is a problem with the com.sun.jndi.ldap code. Decompiling the com.sun.jndi.ldap.Connection class (jdk1.5.0_12) I see this:

BerDecoder readReply(LdapRequest ldaprequest) throws IOException, NamingException
{
_L2:
    BerDecoder berdecoder;
    if((berdecoder = ldaprequest.getReplyBer()) != null)
    break; /* Loop/switch isn't completed */
    try
    {
label0:
    {
        synchronized(this)
        {
        if(sock == null)
            throw new ServiceUnavailableException((new StringBuilder()).append(host).append(":").append(port).append("; socket closed").toString());
        }
        synchronized(ldaprequest)
        {
        berdecoder = ldaprequest.getReplyBer();
        if(berdecoder == null)
        {
            ldaprequest.wait(15000L);
            break label0;
        }
        }
        break; /* Loop/switch isn't completed */
    }
    }
    ...

There is apparently a hardcoded timeout of 15000 milliseconds.

Do anyone have any ideas for a fix/workaround?

Answer 1

Sounds similar to this bug, have you tried inspecting the network traffic with packet sniffer to check for this condition?

Answer 2

You're using an old jdk1.5 (jdk1.5.0_12).

I'm having the same issue with jdk1.5_16 using tomcat 5.5. We have a thread which is waiting for a ldap response and it blocks all other thread because I don't know about JBoss but in tomcat at least all ldap authentication are done sequentially.

If you look in the decompiled code you pasted, after the wait 15 second you have a break label0 which is effectively a loop. So it will loop until ldap answer (no timeout!).

I'm not sure in which version it was fixed, but in 1.5.0_22 the code is now:

BerDecoder readReply(LdapRequest paramLdapRequest)
  throws IOException, NamingException
{
   BerDecoder localBerDecoder;
   int i = 0;

   while (((localBerDecoder = paramLdapRequest.getReplyBer()) == null) && (i == 0)) {
     try
     {
       synchronized (this) {
         if (this.sock == null) {
           throw new ServiceUnavailableException(this.host + ":" + this.port + "; socket closed");
         }
       }

       synchronized (paramLdapRequest)
       {
         localBerDecoder = paramLdapRequest.getReplyBer();
         if (localBerDecoder == null)
           if (this.readTimeout > 0)
           {
             paramLdapRequest.wait(this.readTimeout);
             i = 1;
           } else {
             paramLdapRequest.wait(15000L);
           }
         else
           break label163:
       }
     }
     catch (InterruptedException localInterruptedException) {
       throw new InterruptedNamingException("Interrupted during LDAP operation");
     }

 }

So now, if you provide a timeout value it will wait for that time, and then exit the loop. An this should unlock the authentication queue.