Synchronize Redis with Postgresql

Question

I have a use case where the persistent data would remain in PostgreSQL, and lookups happen through a Redis instance. I would like to do a frequent one-way synchronization from PostgreSQL to Redis to keep the Redis instance up-to-date. (It is okay for Redis to have stale data for a short period of time.)

Is there any tool, library or technique to do this (I need not reinvent the wheel)?

My application stack has PostgreSQL and Ruby on Rails. Node.js can also be considered.

Answer 1

It depends on your time constraints. Is it real-time? +- 1 second? +- 1 minute? +- 1 day?

If you are looking for the fastest solution (< 1 second), I recommand looking at Storm: a topology with high computation power which would dirty check your database (PostGreSQL here with a "Spout") on tables you have defined and would push the results in redis (in a "Bolt"). However, it is a lot overpowered but if you have many evolutions, it would be more flexible than basic scripts and it should be quite resilient and scalable.

http://storm.incubator.apache.org/

If you can "wait" a little longer, well you could just define a periodic lua script that will check PostGreSQL and put the newer values in redis. Lua is good as there are clients for PostGreSQL and Redis but many other languages will fit too. Also Lua can be used inside redis with "eval" command so it should be good to use the same language.

You could also look at ETL solutions, there are many.

Be careful with sync: Redis has optimistic locking only, if you need pessimistic (contention on network etc...) either you define your own lock (there are examples in "Redis in Action" book), either you use a Lua script (Redis is monothreaded: when a lua script is executed, nothing else is executed concurrently).

If you don't like dirty checks, take a look at PostGreSQL triggers and see if you can define some and call scripts from them.