Opening redis connection is too slow

Question

It sometimes takes very long time to open connection to Redis. Looks like it depends on connecting thread's count and, maybe, PC configuration. I run test for 50 threads on two workstations with 4-core CPU's, and it takes 70-100ms to open connection, and on 8-core workstation and 8-core staging server it took 1000-1500ms and sometimes much more. Strange dependency, but it' reproductible. When IIS application pool restarts, and all threads are trying to reconnect, it causes something like cache downtime. What I have to change to get reasonable connection time?

I use BookSleeve client, and here is code sample:

static void Main(string[] args)
{
    for (var i = 0; i < threadCount; i++)
        threads.Add(new Thread(RunThread));

    foreach (var thread in threads)
        thread.Start();

    foreach (var thread in threads)
        thread.Join();
}

static void RunThread()
{
    var connection = TryGetConnection();
    while (connection == null)
    {
        connection = TryGetConnection();
    }
}

static RedisConnection TryGetConnection()
{
    var connection = currentConnection;
    if ((connection != null) && (connection.State == RedisConnectionBase.ConnectionState.Open))
        return connection;

    lock (syncRoot)
    {
        if ((currentConnection != null) && (currentConnection.State == RedisConnectionBase.ConnectionState.Open))
            return currentConnection;

        if ((connectionTask != null) && connectionTask.IsCompleted)
            connectionTask = null;

        if (connectionTask == null)
        {
            if ((currentConnection != null) && (currentConnection.State == RedisConnectionBase.ConnectionState.Closed))
            {
                currentConnection.Dispose();
                currentConnection = null;
            }

            if (currentConnection == null)
            {
                currentConnection = new RedisConnection(
                    serverAddress,
                    serverPort,
                    ioTimeout: (int) operationTimeout.TotalMilliseconds,
                    syncTimeout: (int) operationTimeout.TotalMilliseconds);
            }

            if (currentConnection.State == RedisConnectionBase.ConnectionState.New)
                currentConnection.Open();
        }
    }
    return null;
}

Answer 1

Let's look; we have a loop here:

var connection = TryGetConnection();
while (connection == null)
{
    connection = TryGetConnection();
}

it is not clear to me that TryGetConnection is correctly handling all scenarios ("opening", etc), but frankly, it is a moot point : if you are going to do a tight loop until you get a connection, you might as well simplify significantly. The first thing you could do would be to wait on the task (with a timeout, obviously), rather than using a hot loop. Generalizing:

var task = connection.Open();
connection.Wait(task);

The Wait in the above uses the connection's specified timeout, and does some work to simplify exceptions for you.

However, in this specific case, you could probably just use something like:

var connection = TryGetConnection();
// no loop here

with:

static RedisConnection TryGetConnection()
{
    var connection = currentConnection;
    if ((connection != null) && (connection.State == RedisConnectionBase.ConnectionState.Open))
        return connection;

    lock (syncRoot)
    {   // double-checked
        if ((connection != null) && (connection.State == RedisConnectionBase.ConnectionState.Open))
            return connection;

        connection = ConnectionUtils.Connect(config);
        return connection;
    }
}

where config is a composite of the values; basically something like:

myserver:6389,syncTimeout=1000

This configuration string can also be more complex, including multiple redis servers (master/slave etc), a service-name (for use with sentinel), a client-name (for use with client list), etc.

I suspect some of the complexity in your method is leading to some extra looping at the moment. See if it is any more reliable with the above.