How to properly handle Redis connections with Python / Python RQ?

Question

What is the best pattern to handle Redis connections (both for interacting with Redis directly and indirectly through Python-RQ)?

Generally, database connections need to be closed / returned to a pool when done, but I don't see how to do that with redis-py. That makes me wonder if I'm doing it the wrong way.

Also, I have seen some performance dips when enqueuing jobs to RQ, which I've been told might relate to poor connection usage / reuse.

Basically, I'm interested in knowing the correct pattern, so I can either verify or correct what we have in our application.

Thanks a lot! If there's more information that would be helpful, let me know.

Answer 1

Behind the scenes, redis-py uses a connection pool to manage connections to a Redis server. By default, each Redis instance you create will in turn create its own connection pool. You can override this behavior and use an existing connection pool by passing an already created connection pool instance to the connection_pool argument of the Redis class. You may choose to do this in order to implement client side sharding or have finer grain control of how connections are managed

pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
r = redis.Redis(connection_pool=pool)