RedisTimeoutException is crashing my aspnet core application

Question

When my application traffic gets high, StackExchange.Redis starts to throw RedisTimeoutException and after some minutes, my asp.net core application crashes.
The Windows event viewer says The process was terminated due to an unhandled exception. Exception Info: StackExchange.Redis.RedisTimeoutException. Ok, I understand that there is some issue between my app and Redis, but while I can't solve this, how can I prevent the application to shutdown?

Inside startup.cs, I tried to put:

TaskScheduler.UnobservedTaskException += (object sender, UnobservedTaskExceptionEventArgs eventArgs) =>
{
    eventArgs.SetObserved();
    eventArgs.Exception.Handle(ex => true);
};

no success....

Any help ?

Tks

Answer 1

I agree with @thepirat000's answer, reason is ConnectionMultiplexer

You can use ConnectionMultiplexer according your Redis package (StackExchange.Redis or ServiceStack.Redis) and according your deployment environment

In my aspnet core application (like you) i have used StackExchange.Redis and i have deployed to windows server without any error within below Startup.cs settings

        #region Redis settings ConnectionMultiplexer
        services.AddDataProtection().ProtectKeysWithDpapi(protectToLocalMachine: true);
        services.AddDataProtection()
            .PersistKeysToFileSystem(new DirectoryInfo(@"c:\temp-keys"))
            .ProtectKeysWithDpapiNG($"CERTIFICATE=HashId:{thumbPrint}", flags: Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiNGProtectionDescriptorFlags.None);
        services.AddDataProtection().ProtectKeysWithDpapiNG();

        services.Configure<StorageConfiguration>(new ConfigurationBuilder().SetBasePath(Directory.GetCurrentDirectory()).AddJsonFile("appsettings.json", optional: true, reloadOnChange: true).Build());
        var redisConf = Configuration.GetSection("RedisConnection").Get<RedisConnection>();
        ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(redisConf.Host.ToString() + ":" + redisConf.Port.ToString());
        services.AddDataProtection().PersistKeysToStackExchangeRedis(redis, "DataProtection-Keys");
        services.AddSingleton<IConnectionMultiplexer>(ConnectionMultiplexer.Connect(redisConf.Host.ToString() + ":" + redisConf.Port.ToString()));
        #endregion

Look here for basic usage https://stackexchange.github.io/StackExchange.Redis/Basics.html

Answer 2

Have you tried to put the block that throws the exception in a try/catch block? And perhaps make it try a few times with Polly when there is a timeout. https://github.com/App-vNext/Polly

Normally it shouldn't terminate your app, but since you didn't share any code, We can not be sure.

If you create a service class like below, you can encapsulate all of your redis calls, therefore catch the exceptions.

public class EmptyClass
{
    private readonly ConnectionMultiplexer _connectionMultiplexer;

    public EmptyClass(ConnectionMultiplexer connectionMultiplexer)
    {
        _connectionMultiplexer = connectionMultiplexer;
    }

    public void Execute(Action<ConnectionMultiplexer> action)
    {
        try
        {
            action.Invoke(_connectionMultiplexer);
        }
        catch(RedisTimeoutException ex)
        {

        }
    }

    public void TestRun()
    {
        Execute((ConnectionMultiplexer obj) =>
        {
            //do stuff with obj.
        });
    }
}