How to make sure my event is handled by only one instance of my app?

Question

In our architecture we have a Redis server we use for caching and for publishing event.

My problem is the following

• I have an message called "CustomerUpdate"
• I have 1 application listening to this message
• 3 instance (server) of this application are being executed for scalability
• 1 instance of the database is running
• One of the handler for this message will update the database
• Some other handler will erase memory cache or do something local to the instance

Is there any pattern for making sure that the database is not updated by every instance of the application?

Answer 1

You can use a redis key/value as blocker. When instances receive message from subcription excute LUA script in redis to check if process already exist for it.

Server receive message from subscription Use redis script transaction to check if there already exist a lock for this message (something Like get receiveMessageId:XXX). If value already exit with false then do nothing on server. If the value doesn't exist set it and return true. Then your server can process the message.

As Redis is single threaded all other server will get a false if message is taken by an other server.

To remove this key you can set a TTL big enought to avoid taken message from other servers.

Answer 2

A much simpler idea

1) Instead of publishing your events to a channel "CustomerUpdate" put them in a queue with a unique notifier notifying the type of action it is

LPUSH CustomerUpdate type1$somework

Here type1 can be sending email, entry in db, etc. and somework is the work that you need to handle.

2) In your application logic, use a blocking rpop.

BRPOP CustomerUpdate

in your app logic split the type of work and the work. if it returns type1 do that action, if it return type2 do that and so on. Then carry that work.

Sample snippet:

String message = jedis.brpop("CustomerUpdate",1000);
if(message.startsWith("type1$"))
sendMail(message.split("$")[1]);
else if(message.startsWith("type2$"))
sendAck(message.split("$")[1]);

Pros:

• No need of keyspace notifications
• Even if the app server is down you can do the works as in queue when it gets restarted, Whereas in pub/sub you can't get the messages already published.
• Even if redis is down for few secs, you can get the data when persistence is enabled
• Simple datastructre ( just a single queue )

Answer 3

A simplistic way of doing it, rather than sending in the message the event data, send the name of the list that contains such data, then the first receiver of the message will execute a LPOP on such list and only it will receive the event data.

In a nutshell:

• your clients subscribe like SUBSCRIBE CustomerUpdate.
• the publisher executes RPUSH CustomerUpdateList <data>; PUBLISH CustomerUpdate CustomerUpdateList.
• all the clients will get a MESSAGE CustomerUpdate CustomerUpdateList, but only the first one executing LPOP CustomerUpdateListwill get the message <data>.

However, from the moment you execute the LPOP in the server, the message will be processed or will be lost forever. If for example the connection drops right after the LPOP, the message will be lost.

Implementing reliable messaging in Redis is hard, so you may be better taking a look at projects like : https://github.com/resque/resque or https://github.com/seomoz/qless

Or if you want to do it yourself, take a look to this presentation where the authors give a good explanation about the approach they followed : https://www.percona.com/news-and-events/percona-university-smart-data-raleigh/using-redis-reliable-work-queue

PS: Although my recommendation would be to get something like RabbitMQ for this kind of things.