redis for logging

Question

I am thinking of using Redis for web application logging purposes. I googled that there are people using this approach by dumping the logs into a Redis queue/list, and then a scheduled worker to write to disk.

http://nosql.mypopescu.com/post/8652869828/another-redis-use-case-centralized-logging

I wish to seek understanding that why not directly use Redis to persist to disk? If I have allocated a small server which Redis will write to, separated from the database, app server, is it feasible to use Redis to persist the logs directly?

I also need help in querying Redis by datetime, user, etc. For example, each log is as follow.

datetime=>2012-03-24 17:45:12
userid=>123
message=>test message
category=>my category

How can I query for results within a datetime range, by a specific user, of a particular category?

Thanks!

Answer 1

Redis is in memory datastore. Direct persistence of data to disk is possible with Save or BGSAVE command. Persistence (RDB/AOF) is a feature in addition to storage in-memory.

Requirement mentioned is to store logs to disk. Using any of message queues (like RabbitMQ) instead of in-memory datastore should make things simple. (logs will not eat-up memory)

Applications generating logs can publish them on queues and with separate consumers consuming log messages and writing them to disk.

How can I query for results within a datetime range, by a specific user, of a particular category?

Every block of log should be saved as a structure (example for C/C++) something like this:

   struct log{
     long datatime;
     string userId;
     string message;
     string category;
   };

Serialize this structure to string and store it in Redis as value. Keys for such values would be like: key = userId + DELIMITER + category + DELIMITER + datatime

You can have function which gets all the keys back and split them to get list of data for your specific keyword.