Redis lua when to really use it?

Question

I've started to research and play a bit with lua and have found it to be great when wanting to grab ranges of keys. Ex:

business:5:visits:2013-11-12
business:5:visits:2013-11-13
etc

With lua I only have to send one command to redis instead of the complete range of dates.

Now I'm thinking about converting more of our logic and move it onto Redis.

Take our message storing process which currently looks like this:

// create a new unique id
redisClient.incr(Config.messageId, function(err, reply) {
    var messageId = reply.toString();
    var timestmp = Date.now();

    // push message
    redisClient.zadd(Config.history + ':' + obj.uid + ':' + obj.channel.replace(/\s+/g, ''), timestmp, messageId);

    // store the message data by messageId
    redisClient.hmset(Config.messageHash + ':' + messageId, {
        'user_id': obj.uid,
        'text_body': "some text",
        'text_date': timestmp,
    });


    // set expires
    redisClient.expire(Config.history + ':' + obj.uid + ':' + obj.channel.replace(/\s+/g, ''), Config.messageExpire);
    redisClient.expire(Config.messageHash + ':' + messageId, Config.messageExpire);


    // add to mysql-sync queue
    redisClient.RPUSH(Config.messageMySQLSyncQueue, Config.messageHash + ':' + messageId);

});

The above could easily be converted into lua, but is it worth it for performance?

Would it be faster to write this in Lua instead and only have to issue 1 command to Redis? Would it cause problems with blocking other commands?

Answer 1

Lua scripts are meant to work like MULTI commands. Actually most commands that you would develop using MULTI commands from a Redis client can be implemented in Lua. That is, you can encapsulate some complex operations in a script and your data layer will perform the atomic write operation without worrying about your data modeling strategy on Redis.

Also, I find them useful when you want to perform quick but complex read operations. For example, you might want to get objects in order. Objects are stored in a hash key while the order is defined by a sorted set key. You get a range of the so-called sorted set and you get objects in hash using hmget.

The most important point is Lua scripts should implement things that can execute as fast as possible, since Redis will block other operations while a Lua script is running. That is, you need to perform quick interruptions or your overall Redis performance will decrease a lot.

Arguments to not use Lua

I would argue that you should use them when you really need them. Usually, clients are developed using high-level programming languages like C#, Java, JavaScript, Ruby... and they provide better development experience: good debuggers, IDE, code-completion...

Summary: you should use them if you can prove that there's an actual benefit (in performance) if you turn some part of your domain logic into Redis Lua scripts.

Answer 2

There is one more drawback of Redis's Lua scripts: you can't issue operations to another instances of Redis even though they're on the same physical server. It's a common practice to run multiple instances of Redis on one server to utilize all the CPU cores. So you have to control many instances of Redis from a client.

To summarize everything that was said above:

1. Lua scripts are pure functions. You can't do outside requests like http requests or even requests to other instances of Redis. You can't make any side effects.
2. Lua scripts block everything. You can't run two Lua scripts in parallel. As a consequence you can't do any big job in Lua e.g. you can't run an infinite loop in Lua to do something in background. It's like to format a floppy disk on Windows 95 if you know what I mean :-)

In short, Redis is not an application server. So you cannot easily write any logic you want on Lua and be sure that everything is alright.

If you need a real application server inside a NoSQL database then try Tarantool for example. It also has Lua scripts, but the difference is that they do not block each other, they're still executed as a one ACID transaction (if you don't do external requests) and they may do any side effects you want, even issue requests to external databases like Tarantool or Redis or do any http requests. Which is achieved by executing each Lua script in a separate fiber and by row-based replication of all the changes that are done by Lua scripts.