NHibernate & WCF: Performance (session reuse) vs. concurrency (simultaneous requests)

Question

We are working on different integrations to a swarm of identically structured legacy databases that basically cannot be altered. For this, we added an auxiliary database for holding things like meta-information, routing rules and for temporarily holding data for the legacy databases.

We are mainly using NHibernate to connect to the databases. One application is a WCF Service that needs to inserts incoming data into nested tables that are really wide (dozens of columns). Obviously, performance is an issue, so I have been looking to be as economic as possible with the NHibernate transactions. At the same time, concurrency appeared to be an issue. In production, we were starting to get some zombied transaction errors (deadlocks).

I have been doing a balancing act to deal with these two issues but haven't really eliminated the concurrency problems.

The service behaviour is set to handle one request at a time, like so:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall, ConcurrencyMode=ConcurrencyMode.Single]
public class LegacyGateService : ILegacyGateService

Earlier, after some "inspiration" (read: copy/paste) from the internet, I ended up adding a set of classes called something like XxxNHibernateUtil, for the auxiliary database and the legacy databases, respectively. These classes control the NHibernate Sessions and generate or re-use the Sessions from pre-initialized SessionFactories.

For the Auxiliary database it looks like this:

public static class LegacyGateNHibernateUtil
{
    private static readonly ISessionFactory sessionFactory = BuildSessionFactory();

    private static ISessionFactory BuildSessionFactory()
    {
        try
        {
            Configuration Cfg = new Configuration();
            Cfg.Configure();
            Cfg.AddAssembly("LegacyGate.Persistence");
            return Cfg.BuildSessionFactory();
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }

    public static ISessionFactory GetSessionFactory()
    {
        return sessionFactory;
    }

    public static ISession GetCurrentSession()
    {
        if (!CurrentSessionContext.HasBind(GetSessionFactory()))
            CurrentSessionContext.Bind(GetSessionFactory().OpenSession());

        return GetSessionFactory().GetCurrentSession();
    }

    public static void DisposeCurrentSession()
    {
        ISession currentSession = CurrentSessionContext.Unbind(GetSessionFactory());

        if (currentSession != null)
        {
            if (currentSession.IsOpen)
                currentSession.Close();
            currentSession.Dispose();
        }
    }
}

Whenever a session is needed for a transaction, the current session is looked up and re-used for the duration of the service request call. Or at least: That is what is supposed to be happening.

EDIT: The session context is of course set in the hibernate.cfg.xml like so:

  <property name="current_session_context_class">call</property>

For the legacy databases, the NHibernateUtil is adapted to deal with different possible databases. For this, each connection gets its own SessionFactory built that has to be looked up in a Dictionary collection. Otherwise, the principles are the same.

Testing using WCFStorm, this seems to be working fine when sending one request at a time, but as soon I start a load test, even with just one agent and long intervals, I get a plethora of different kinds of exceptions all pointing to simultaneous requests and transactions sabotaging each other. I have tried tweaking the IsolationLevel, but of now avail.

I think I need to generate and handle Sessions in a different way, so that different transactions to the same databases are handled in an orderly fashion and not interfere with each other. However, I lack some insight in how to make this work. Any help is greatly appreciated!

---

EDIT For one service method, when testing with more than one agent, the first dozen or so call work fine, and then the following string of exceptions start appearing that only pertain the auxiliary database:

1. "There was a problem converting an IDataReader to NDataReader" / "Invalid attempt to call MetaData when reader is closed."
2. "illegal access to loading collection"
3. "Begin failed with SQL exception" / "Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding."
4. "could not execute query" / "ExecuteReader requires an open and available Connection. The connection's current state is closed."
5. "could not initialize a collection:" / "Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding."
6. "Transaction not successfully started"
7. "The transaction is either not associated with the current connection or has been completed."
8. "could not initialize a collection:" / "Invalid attempt to call Read when reader is closed."

Exception 1, at least, indicates that the same session is accessed by multiple threads (calls probably). Also the other ones indicate that the current session is interrupted by other processes. But how can this be, when I tried to isolate the calls and have them queued up?

For another service method, these issues do not appear with the auxiliary database, but after some time I start getting the ZombiedTransaction exceptions (deadlocks) with transactions to the legacy databases. Still... What gives?

Answer 1

The easy answer: You don't re-use NHibernate sessions.

They're not heavyweight objects and they are designed to be created, manipulated and disposed following the Unit of Work pattern. Attempting to "share" these across multiple requests goes against their intended usage.

Fundamentally, the cost of synchronizing access to the sessions correctly will almost certainly negate any benefit you gain by recycling them to avoid re-initializing them. Let's also consider that these costs are a drop in the pond of the SQL you'll be executing.

Remember that NHibernate's Session Factory is the heavyweight object. It's thread-safe, so you can and should share a single instance across all requests out of the box.

Your code should look conceptually like this:

public class Service : IService
{
    static Service()
    {
        Configuration Cfg = new Configuration();
        Cfg.Configure();
        Cfg.AddAssembly("LegacyGate.Persistence");
        Service.SessionFactory = Cfg.BuildSessionFactory();
    }

    protected static ISessionFactory SessionFactory { get; private set; }

    public void ServiceMethod(...)
    {
        using(var session = Service.SessionFactory.CreateSession())
        {
            // Do database stuff
            ...
        }
    }
}

As an aside: Ideally, you'd be dependency-injecting the ISessionFactory into the service.