Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Redis distributed increment with locking

I have a requirement for generating an counter which will be send to some api calls. My application is running on multiple node so some how I wanted to generate unique counter. I have tried following code

public static long GetTransactionCountForUser(int telcoId)
{
    long valreturn = 0;
    string key = "TelcoId:" + telcoId + ":Sequence";
    if (Muxer != null && Muxer.IsConnected && (Muxer.GetDatabase()) != null)
    {
        IDatabase db = Muxer.GetDatabase();
        var val = db.StringGet(key);
        int maxVal = 999;
        if (Convert.ToInt32(val) < maxVal)
        {
            valreturn = db.StringIncrement(key);
        }
        else
        {
            bool isdone = db.StringSet(key, valreturn);
            //db.SetAdd(key,new RedisValue) .StringIncrement(key, Convert.ToDouble(val))
        }
    }
    return valreturn;
}

And run tested it via Task Parallel libray. When I have boundary values what i see is that multiple time 0 entry is set

Please let me know what correction i needed to do

Update: My final logic is as following

public static long GetSequenceNumberForTelcoApiCallViaLuaScript(int telcoId)
{
    long valreturn = 0;
    long maxIncrement = 9999;//todo via configuration
    if (true)//todo via configuration
    {
        IDatabase db;
        string key = "TelcoId:" + telcoId + ":SequenceNumber";
        if (Muxer != null && Muxer.IsConnected && (db = Muxer.GetDatabase()) != null)
        {
            valreturn = (long)db.ScriptEvaluate(@"
                local result = redis.call('incr', KEYS[1])
                if result > tonumber(ARGV[1]) then
                result = 1
                redis.call('set', KEYS[1], result)
                end
                return result", new RedisKey[] { key }, flags: CommandFlags.HighPriority, values: new RedisValue[] { maxIncrement });
        }
    }
    return valreturn;
}
like image 494
Kamran Shahid Avatar asked Jan 19 '16 08:01

Kamran Shahid


People also ask

How does distributed locking work in Redis?

With distributed locking, we have the same sort of acquire, operate, release operations, but instead of having a lock that's only known by threads within the same process, or processes on the same machine, we use a lock that different Redis clients on different machines can acquire and release.

Does Redis handle locking?

As long as the majority of Redis nodes are up, clients are able to acquire and release locks.

Is Redis increment Atomic?

"You might not know it, but Redis is actually single-threaded, which is how every command is guaranteed to be atomic. While one command is executing, no other command will run."

What is Redis RedLock?

Redlock - A ruby distributed lock using redis. Distributed locks are a very useful primitive in many environments where different processes require to operate with shared resources in a mutually exclusive way.


1 Answers

Indeed, your code is not safe around the rollover boundary, because you are doing a "get", (latency and thinking), "set" - without checking that the conditions in your "get" still apply. If the server is busy around item 1000 it would be possible to get all sorts of crazy outputs, including things like:

1
2
...
999
1000 // when "get" returns 998, so you do an incr
1001 // ditto
1002 // ditto
0 // when "get" returns 999 or above, so you do a set
0 // ditto
0 // ditto
1

Options:

  1. use the transaction and constraint APIs to make your logic concurrency-safe
  2. rewrite your logic as a Lua script via ScriptEvaluate

Now, redis transactions (per option 1) are hard. Personally, I'd use "2" - in addition to being simpler to code and debug, it means you only have 1 round-trip and operation, as opposed to "get, watch, get, multi, incr/set, exec/discard", and a "retry from start" loop to account for the abort scenario. I can try to write it as Lua for you if you like - it should be about 4 lines.


Here's the Lua implementation:

string key = ...
for(int i = 0; i < 2000; i++) // just a test loop for me; you'd only do it once etc
{
    int result = (int) db.ScriptEvaluate(@"
local result = redis.call('incr', KEYS[1])
if result > 999 then
    result = 0
    redis.call('set', KEYS[1], result)
end
return result", new RedisKey[] { key });
    Console.WriteLine(result);
}

Note: if you need to parameterize the max, you would use:

if result > tonumber(ARGV[1]) then

and:

int result = (int)db.ScriptEvaluate(...,
    new RedisKey[] { key }, new RedisValue[] { max });

(so ARGV[1] takes the value from max)

It is necessary to understand that eval/evalsha (which is what ScriptEvaluate calls) are not competing with other server requests, so nothing changes between the incr and the possible set. This means we don't need complex watch etc logic.

Here's the same (I think!) via the transaction / constraint API:

static int IncrementAndLoopToZero(IDatabase db, RedisKey key, int max)
{
    int result;
    bool success;
    do
    {
        RedisValue current = db.StringGet(key);
        var tran = db.CreateTransaction();
        // assert hasn't changed - note this handles "not exists" correctly
        tran.AddCondition(Condition.StringEqual(key, current));
        if(((int)current) > max)
        {
            result = 0;
            tran.StringSetAsync(key, result, flags: CommandFlags.FireAndForget);
        }
        else
        {
            result = ((int)current) + 1;
            tran.StringIncrementAsync(key, flags: CommandFlags.FireAndForget);
        }
        success = tran.Execute(); // if assertion fails, returns false and aborts
    } while (!success); // and if it aborts, we need to redo
    return result;
}

Complicated, eh? The simple success case here is then:

GET {key}    # get the current value
WATCH {key}  # assertion stating that {key} should be guarded
GET {key}    # used by the assertion to check the value
MULTI        # begin a block
INCR {key}   # increment {key}
EXEC         # execute the block *if WATCH is happy*

which is... quite a bit of work, and involves a pipeline stall on the multiplexer. The more complicated cases (assertion failures, watch failures, wrap-arounds) would have slightly different output, but should work.

like image 169
Marc Gravell Avatar answered Sep 23 '22 13:09

Marc Gravell