@Asynchronous EJB and hibernate connection issues

Question

    @Stateless
    public class MyStatelessBeanA {
        @Resource
        private SessionContext sessionCtx;  

       public byte[] methodA(){
          MyStatelessBeanA myStatelessProxy1 = this.sessionCtx.getBusinessObject(MyStatelessBeanA.class);
          MyStatelessBeanA myStatelessProxy2 = this.sessionCtx.getBusinessObject(MyStatelessBeanA.class);

          Future<byte[]> proxy1Future = myStatelessProxy1.asynchMethod();
          Future<byte[]> proxy2Future = myStatelessProxy2.asynchMethod();

          byte[] firstArray = proxy1Future.get();
          byte[] secondArray = proxy2Future.get();

          return ...

       }

       @Asynchronous
       public Future<byte[]> asynchMethod(){
           byte[] byteArray = ...
           ...do something including select from various table...
           return new AsynchResult<byte[]>(byteArray);
       }

Basically what i'm trying to do is to call asynchMethod() twice but in parallel from the two proxy objects.

The issue?

2019-11-05 17:20:23,354 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 3) SQL Error: 0, SQLState: null
2019-11-05 17:20:23,354 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 3) IJ031041: Connection handle has been closed and is unusable
2019-11-05 17:20:23,354 INFO  [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (EJB default - 2) IJ000311: Throwable from unregister connection: java.lang.IllegalStateException: IJ000152: Trying to return an unknown connection: org.jboss.jca.adapters.jdbc.jdk7.WrappedConnectionJDK7@69ebfad0
    at org.jboss.jca.core.connectionmanager.ccm.CachedConnectionManagerImpl.unregisterConnection(CachedConnectionManagerImpl.java:408)
    at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.connectionClosed(TxConnectionListener.java:645)
    at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.returnHandle(BaseWrapperManagedConnection.java:617)
    at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.closeHandle(BaseWrapperManagedConnection.java:562)
    at org.jboss.jca.adapters.jdbc.WrappedConnection.returnConnection(WrappedConnection.java:298)
    at org.jboss.jca.adapters.jdbc.WrappedConnection.close(WrappedConnection.java:256)
    at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.closeConnection(DatasourceConnectionProviderImpl.java:127)
    at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.releaseConnection(AbstractSessionImpl.java:397)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.releaseConnection(LogicalConnectionManagedImpl.java:172)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.afterStatement(LogicalConnectionManagedImpl.java:125)
    at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.afterStatementExecution(JdbcCoordinatorImpl.java:281)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:145)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:86)
    at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:167)
    at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3956)
    at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:508)
    at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:478)
    at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219)
    at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:116)
    at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89)
    at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1129)
    at org.hibernate.internal.SessionImpl.immediateLoad(SessionImpl.java:997)
    at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:157)
    at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:266)
    at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:68)

The two calls for asynchMethod() are correctly assigned to two different threads:

2019-11-05 17:20:22,566 INFO  [**.*******.*******.*******.MyStatelessBeanA] (EJB default - 2) method=asynchMethod START
2019-11-05 17:20:22,655 INFO  [**.*******.*******.*******.MyStatelessBeanA] (EJB default - 3) method=asynchMethod START

Is it possible that in someway one proxy object close the other one's connection? I don't know if it is enough information to guess a proper solution to the problem but i'm looking everything possible (CachedConnectionManagerImpl source code,TxConnectionListener source code) but seems something beyond my skills.

If anyone can help or give some hint as i'm completely stuck on this.

Thanks, Davide

ADDED INFORMATION THAT MAY BE USEFUL

Standalone.xml hibernate part

        <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
        <local-cache name="entity">
            <transaction mode="NON_XA"/>
            <eviction strategy="LRU" max-entries="10000"/>
            <expiration max-idle="100000"/>
        </local-cache>
        <local-cache name="local-query">
            <eviction strategy="LRU" max-entries="10000"/>
            <expiration max-idle="100000"/>
        </local-cache>
        <local-cache name="timestamps"/>
    </cache-container>

Persistence.xml

<?xml version="1.0" encoding="UTF-8" ?>
<persistence version="2.0"
    xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">


        <persistence-unit name="***" transaction-type="RESOURCE_LOCAL">

            <jar-file>file:./target/test-classes</jar-file>

            <exclude-unlisted-classes>false</exclude-unlisted-classes>

            <properties>
                <property name="hibernate.archive.autodetection" value="class,hbm" />
                <property name="hibernate.connection.url" value="${dbunit.connectionUrl}" />
                <property name="hibernate.connection.driver_class" value="${dbunit.driverClass}" />
                <property name="hibernate.dialect" value="${dbunit.jpa-dialect}" />
                <property name="hibernate.connection.username" value="${dbunit.username}" />
                <property name="hibernate.connection.password" value="${dbunit.password}" />
                <property name="javax.persistence.validation.mode" value="NONE" />
                <property name="hibernate.show_sql" value="true" />

            </properties>

        </persistence-unit>
    </persistence> 

Answer 1

This isn't a full answer because I've not been able to establish a true root cause, but I'll put down here what I found and a possible fix.

I've encountered this issue in the Wildfly application server (a fairly recent version too at the time of this writing) and seem to recall seeing the same on a GlassFish server quite a while ago. The issue seems to be with Hibernate rather than the EJB container or JTA manager, although I'm not 100% sure of this. Despite the warning and annoying stack traces, the code seems to behave correctly according to your expectations. Transactions get committed and aren't rolled back due to the error, the invocations seem to run with their own connections in separate transactions and I haven't noticed some sort of connection leak that leads to issues after extended periods. So at least functionally things seem fine.

The main suspect here is that somehow the scope in which the connection and transaction are established isn't the same as the scope in which these are closed. I don't know if either the asynchronous invocation or the self-injection of the EJB is what triggers the issue and if one of these is enough to replicate it. My suspicion is that the call to the @Asynchronous method will obtain a database connection for the entity manager and start a transaction and this gets somehow related to the thread or some other context that called the async method. But of course, the method being asynchronous, the actual execution will be handled by the EJB container in a separate thread and the method immediately returns, either returning nothing if its return type is void or returning a Future for obtaining a result. When the method completes the EJB container takes care of transaction commit or rollback and releasing the connection. Hibernate does not seem to like seeing this happen in different contexts or threads, hence the error. But the error occurs after the transaction completed (so the code works) and because it happens in some container-managed thread, not that of the caller, the exception never bubbles up to your code.

Now if that was all there is to it, an asynchronous EJB method could consistently replicate this issue. But it doesn't seem to do that. The self-inject of the EJB also seems essential to trigger this. In fact, in the GlassFish project way back where I encountered this I don't recall using async methods, but I did have an EJB obtain another instance of its own class. So maybe that alone is sufficient to get this behaviour. Why it would happen in that case is something I haven't managed to figure out, but I'll come back to it after an example of a possible solution.

Regardless of the cause, a workaround I've used that seems to get rid of the error is to simply use an @EJB injection in the EJB class itself rather than going through a @Resource-injected SessionContext. So your code could be rewritten as follows:

@Stateless
public class MyStatelessBeanA {
    @EJB
    private MyStatelessBeanA myStatelessProxy1;
    @EJB
    private MyStatelessBeanA myStatelessProxy2;  

   public byte[] methodA(){
      Future<byte[]> proxy1Future = myStatelessProxy1.asynchMethod();
      Future<byte[]> proxy2Future = myStatelessProxy2.asynchMethod();

      byte[] firstArray = proxy1Future.get();
      byte[] secondArray = proxy2Future.get();

      return ...

   }

   @Asynchronous
   public Future<byte[]> asynchMethod(){
       byte[] byteArray = ...
       ...do something including select from various table...
       return new AsynchResult<byte[]>(byteArray);
   }

This may seem a bit scary because it feels like it would be cause for some infinite recursion of injections. But that doesn't happen. Remember that a field annotated with @EJB or @Inject in some EJB or managed bean is actually a proxy object, and the lookup or instantiation of an actual instance of your class is done when that proxy is used. Those myStatelessProxy1 and myStatelessProxy2 fields are just proxy placeholders and once a method is invoked on them, an instance of the MyStatelessBeanA is assigned to the proxy, has its own injection done and is given resources like a database connection and transaction context. And that instance in turn may have these two fields, but those are also proxies and provided they aren't used, no further recursion occurs. You could accidentally create some endless recursion if the method you invoke on an injected instance relies on one of its own injected fields and then in turn invokes a method on it that does the same. Because this is different from recursion within the same object instance it might not be immediately obvious and a compiler or IDE might not warn you about it, so beware.

This self-injection should be supported starting from the EJB 2.1 API onwards. What's peculiar is that doing this seems to result in the Hibernate error no longer being thrown. To get back to what I mentioned earlier about this, a possible cause could be the point at which the EJB proxy is created. In one case we ask a SessionContext to create an instance, which happens within a method of the EJB. In another case we obtain an instance through an @EJB injection point, which would be done before the EJB method is invoked. Those are two different contexts. One is within your code, maybe with its own database connection and possibly running in a transaction. The other is within the container's code, outside the transaction boundary and possibly before a database connection is established. Now, this being proxies we might expect them to only kick into gear once we use them, but who knows what work the container might be doing upfront when creating the proxy objects and what is handed to them. Maybe current Hibernate versions handle this properly but still trip up when an asynchronous method is thrown into the mix.

The self-injection might solve your problem but wouldn't work if you need a dynamic number of these EJB instances, which could be done when getting them from a SessionContext. But in that case there is probably an issue with the code design anyway. One injection should suffice, because you can reuse that one for multiple asynchronous method invocations. They'd each run in their separate thread. And they'd also have their own transactions. An asynchronous method can't join the transaction of its caller, because then the point at which the caller's transaction completes would no longer be when it exits its transaction boundary. It would complete at "some point" in the future. This messes up the semantics. The only alternative would be to have the caller wait after exiting its method until the async invocations it launched completed, but that would defeat their purpose in some situations (like when you don't need a return value). If you have to spawn an unpredictable number of asynchronous tasks, a ManagedExecutorService might be a better fit. It is part of the concurrency utilities for Java EE, which got introduced with JEE 7.

Finally, another solution could be to just make some extra helper EJB class which gets the other one injected and then uses it. The division may be a bit artificial because it only exists to deal with the effects of container management, not as a direct consequence of your requirements, but it's simple and effective. It's also easy to reason about, contrary to the self-injection, and can assist in writing tests and using mock objects.