Apache load balancer data syncing across multiple applications

Question

I have a setup where two instances of same application deployed behind a load balancer and both of the applications are talking to the same database.

The issue I'm having is that when data inserted into the database via one of the application, the very same data is not accessible/visible to the other application.

These two applications have exactly same configuration in terms of database abstraction framework, spring framework, etc. In addition, these two applications run on different tomcat servers on different physical servers.

What could be the cause to the issue?

Edit

Hibernate is configured as follow:

Edit 2

Here's the ehcache.xml

<cache name="org.hibernate.cache.internal.StandardQueryCache"
       maxElementsInMemory="10000"
       eternal="false"
       timeToLiveSeconds="86400"
       overflowToDisk="false"
       memoryStoreEvictionPolicy="LRU"/>

<defaultCache
        maxElementsInMemory="10000"
        eternal="false"
        timeToLiveSeconds="86400"
        overflowToDisk="false"
        memoryStoreEvictionPolicy="LRU"/>

Answer 1

You are most probably experiencing cache replication issues, probably due to missing/incorrect cache coordination configuration.

Assumption

You have enabled both second level cache and query cache in Hibernate, which is fine of course and you got a performance boost. The problem is that you use 2 nodes, each one having its own cache or they don't invalidate caches correctly.

For example, node A performs a query for all entities Z (i.e. list of product categories: Z1, Z2, ...) and caches the result. Later on, node B adds a new product category entity Zn. Node A will not use that new entity until its own cache is invalidated; until then it will use the cached result set which does not include Zn.

You can verify this assumption if the problem goes away when you disable query caching (i.e. set <property name="hibernate.cache.use_query_cache">false</property>).

Solutions

Option A

Optimize cache invalidation. This greatly depends on your business needs but some suggestions would be:

• Reduce cache TTL for certain queries until you find a sweet spot (time span you can tolerate missing/extra values). You can use EhCache regions for that.
• Disable caching altogether for certain entities

Example:

Create a region for short-lived (i.e. 5 minutes) query results in ehcache.xml:

<cache name = "short-lived"
maxElementsInMemory = "500"
eternal = "false"
timeToIdleSeconds = "600"
timeToLiveSeconds = "600"
overflowToDisk = "false"
/>

For specific queries that need to be frequently refreshed:

Query query = session.createQuery("FROM Z");
query.setCacheable(true);
query.setCacheRegion("short-lived");

More on the topic here.

Option B

This is the real solution. You can have an shared cache for all nodes. While it's definitely harder to setup and configure, it gives you the best possible performance. For example, cached queries will run once per cluster, not once per node as it is now.

Take a look at Terracotta.