How do you *properly* query Redis from Tornado?

Question

I'm curious what the recommended method of querying Redis (or any DB for that matter) is from Tornado.

I've seen some examples like https://gist.github.com/357306 but they all appear to be using blocking calls to redis.

My understanding is that to avoid grinding Tornado to a halt, I need to be using non-blocking DB libraries like the ones developed for Twisted.

Am I wrong? How is this supposed to be done?

Answer 1

When it comes to blocking commands like BLPOP or listening to a Pub/Sub channel you'll need an asynchronous client like tornado-redis. You may start with this demo to see how the tornado-redis client may be used to develope a simple public chat application.

But I would recommend using the synchronous redis-py client in conjunction with hiredis for most other cases.

The main advantage of asynchronous client is that your server can handle incoming requests while waiting for Redis server response. However, the Redis server is so fast that in most cases an overhead of setting up asynchronous callbacks in your Tornado application adds more to the total time of request processing then the time spent on waiting for Redis server response.

Using an asynchronous client you may try to send multiple requests to the Redis server at the same time, but the Redis server is a single-threaded one (just like Tornado server), so it will answer to these requests one-by-one and you'll gain almost nothing. And, in fact, you don't have to send multiple Redis commands at the same time to the same Redis server as long as there are pipelines and commands like MGET/MSET.

An asynchronous client has some advantages when you use several Redis server instances, but I suggest using a synchronous (redis-py) client and a proxy like twemproxy or this one (the latter supports pipelining and MGET/MSET commands).

Also I suggest not to use the connection pooling when using the redis-py client in Tornado applications. Just create a single Redis object instance for each Redis database your application connects to.

Answer 2

I would recommend to use brukva which is an "Asynchronous Redis client that works within Tornado IO loop".

Answer 3

One option is to use the port of Tornado to Twisted and then use the Twisted Redis API with that. Tornado itself doesn't seem to have arbitrary asynchronous operations as an objective (though if you wanted to rebuild all of the sorts of things that have been built for Twisted, you probably could build them from the low-level iostream APIs in Tornado, but I wouldn't recommend it).