Many concurrent reads + one write cause ObjectNotFoundException due to ehcache

Question

I use Hibernate 3.6.8, ehcache 2.4.5 (also tried with latest 2.8.0), jvm 1.6.0_22 on a high traffic site, and sometimes I experience

ObjectNotFoundException: No row with the given identifier exists: [com.example.Foo#123]`

when a new Foo (in this case with id 123) is created via the simplest code possible:

Foo foo = new Foo();
session.save(foo);

The reason is that in all pages of this high traffic site I get all Foos like this:

session.createQuery("from Foo").setCacheable(true).list();

The table storing Foos contains 1000 rows, and the entity is cached in ehcache:

<class-cache class="com.example.Foo" usage="read-write" />

Other possibly relevant parts of my Hibernate configuration are:

<property name="connection.url">jdbc:mysql://localhost:3306/example?characterEncoding=UTF-8</property>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>

<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.idle_test_period">60</property>
<property name="hibernate.c3p0.min_size">10</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.max_statements">0</property>
<property name="hibernate.c3p0.timeout">0</property>
<property name="hibernate.c3p0.acquireRetryAttempts">1</property>
<property name="hibernate.c3p0.acquireRetryDelay">1</property>

<property name="hibernate.show_sql">true</property>
<property name="hibernate.use_sql_comments">true</property>

<property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.jdbc.use_scrollable_resultset">true</property>

<property name="hibernate.cache.provider_class">net.sf.ehcache.hibernate.SingletonEhCacheProvider</property>
<property name="net.sf.ehcache.configurationResourceName">/ehcache.xml</property>
<property name="hibernate.cache.use_query_cache">true</property>

The error happens once and then goes away. I suspect that the ehcache query cache is updated with the new entity id (123) id, but the entity cache is not yet updated with the contents of that entity. I reproduce this fairly easily locally using JMeter.

Any idea on how to solve this?

On Foo creation the ObjectNotFoundException is thrown once. If on the other hand I delete an instance of Foo then I constantly (and forever) get ObjectNotFoundException for each execution of .list(). The stacktrace can be seen at http://pastebin.com/raw.php?i=dp3HBgDB

Answer 1

The read-write strategy does not guarantee transactionality between the database and the cache, so I think this is what happens when the write occurs:

• the new object Foo is attached to the hibernate session of the write.

• a lazy loading proxy is inserted into the second level cache by the hibernate session associated to the write request.

• The new Foo will be inserted in the database by that same session, but the insert will take a certain time to be built, flushed and committed.

• meanwhile another request as hit the proxy to load all Foos. It finds the lazy loading proxy in the cache (see the stacktrace DefaultLoadEventListener.proxyOrLoad()), and decides to load the object (DefaultLoadEventListener.load()).

• This triggers a Hibernate load() of a Foo not yet inserted in the database by the write thread.

• No Foo with that Id is found on the database, so a ObjectNotFoundException is thrown.

To confirm this, put an exception breakpoint on your IDE, to see that at the moment the exception is thrown the object had not yet been inserted in the DB. One way to solve it would be to use the transactional strategy.