Redis: Why is Lua scripting going to replace transactions?

Question

The documentation for transactions says:

"we may deprecate and finally remove transactions" and "everything you can do with a Redis transaction, you can also do with a script"

http://redis.io/topics/transactions

But does it? I see a problem with this.

Within a transaction you can WATCH multiple variables, read those variables, and based on the unique state of those variables you can make a completely different set of writes before calling EXEC. If anything interferes with the state of those variables in the intervening time, EXEC will not perform the transaction. (Which allows you to retry it. This is a perfect transaction system.)

An EVAL script will not let you do that. According to the documentation on this page:

"Scripts as pure functions...The script always evaluates the same Redis write commands with the same arguments given the same input data set. Operations performed by the script cannot depend on any hidden (non explicit) information or state that may change as script execution proceeds or between different executions of the script, nor can it depend on any external input from I/O devices."

http://redis.io/commands/eval

The problem I see with EVAL is that you cannot GET the state of those variables inside of the script and make a unique set of writes based on the state of those variables. Again: "The script always evaluates the same Redis write commands with the same arguments given the same input data set." So the resulting writes are already determined (cached from the first run) and the EVAL script doesn't care what the GET values are inside of the script. The only thing you can do is perform GET for those variables before calling EVAL and then pass those variables to the EVAL script, but here's the problem: now you have an atomicity problem between calling GET and calling EVAL.

In other words, all the variables that you would have done a WATCH for a transaction, in the case of EVAL you instead need to GET those variables and then pass them to the EVAL script. Since the atomic nature of the script isn't guaranteed until the script actually starts, and you need to GET those variables before calling EVAL to start the script, that leaves an opening where the state of those variables could change between the GET and passing them to EVAL. Therefore, the guarantee of atomicity that you have with WATCH you do not have with EVAL for a very important set of use cases.

So why is there talk about deprecating transactions when that would cause important Redis functionality to be lost? Or is there actually a way to do this with EVAL scripts that I do not understand yet? Or are there features planned that can solve this for EVAL? (Hypothetical example: if they made WATCH work with EVAL the same way that WATCH works with EXEC, that might work.)

Is there a solution to this? Or am I to understand that Redis may not be completely transaction safe in the long term?

Answer 1

Its true that lua scripts can do whatever transactions can, but I don't think Redis transactions are going away.

EVAL script does not let you watch variables

When an eval script is running, nothing else can run concurrently. So, watching variables is pointless. You can be sure that nobody else has modified the variables once you have read the values in the script.

The problem I see with EVAL is that you cannot GET the state of those variables inside of the script and make a unique set of writes based on the state of those variables.

Not true. You can pass keys to the eval script. Within your eval script, you can read the values from Redis, and then based on those values conditionally execute other commands.

The script is still deterministic. If you take that script and run it on the slave, it will still execute the same write commands because the master and slave have the same data.

Answer 2

EVAL scripting actually extends and simplifies functionality of transactions.

It may help to view the two types of transactions in Redis in the following way:

1. procedural (MULTI EXEC)

Pure MULTI EXEC is grouping commands to be executed in one go and returns an array of replies from each command. It has a serious limitation. It does not allow using an intermediate result from one command in the next one within the transaction. In this pure form it's not very useful in practical applications. It's basically a pipelining with guaranteed atomicity.

Adding the WATCH key before the transaction, allows for optimistic locking and using in the transaction the values obtained from Redis outside the transaction. If a race condition happens, the transaction fails and must be retried. This complicates application logic and the optimism is often unwarranted since you may end up in the endless retrying loop.

2. functional (EVAL scripting)

EVAL is not only grouping Redis commands but also gives you the power of the full programming language, notably conditions, loops, and local variables. In the Lua script you can read values from Redis in one command and use them later in the next command.

You submit the script which is executed in an atomic way. It's guaranteed by the single threaded, "stop the world" approach. No other script or Redis command will be executed while the script is being executed. Consequently EVAL also has a limitation: scripts must be small and fast to prevent blocking other clients.

We also need to trust that other clients don't submit a slow script which should be considered as a programming error. It suites well into the security model since "Redis is designed to be accessed by trusted clients inside trusted environments".