python, twistedmatrix... logging through socket

Question

I need some help to evaluate what will be the right design of a twisted matrix application. Or any url to help to do so.

• background: i now use logging capacity included with twistedmatrix by using FileLogObserver, and a custom DailyLogFile to propage and save data to file system and for later analyze.

Now, I m going to use many differents such applications that will do theirs jobs and sent log messages as timed event in many files twisted logs (distincts files). Thus my events are composed of (timestamp, data)

I need some way of read each of this logged messages (timestamp, data) in real time and centralize the event grouping them on the time basis, that is having some computations and group by on all the events that has exactly the same timestamp.

am I right to suppose that i can do this with twisted log functionality?

how would you design it in a simple way?

my current thought was to build a socket twisted logger and duplicate the logging processus so that each event will be send to the file logger (i need separated history) but also through the socket to a receiver (i now also need realtime). If I am right digging this way, does someone has a skeleton for such twisted socket logger ? how can i chain two logger from twisted?

for the other part what should be the correct logger receiver?

as i also will propagate the computed/agregated data throught a lighstreamer server using the below scheme can there be some difficulties that I may not have seen, that will need some threading mecanism or others mecanism to avoid some blocking network call?

feed = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
feed.connect(("localhost",MYPORT))

feed.send(mytimestamp, myeventdata)

as a figure it could be schematized as: with (everything almost realtime)

producerTimedEventLog1 --->|
producerTimedEventLog2 --->|
...                        |---> loggerReceiverComputingData ---> lighstreamer process ---> mozilla or whatever webclient
...                        |
producerTimedEventLognN--->| 

each producerTimedEventLognN also logging to a file.

i'm interested in every useful ideas :)

best regards

Answer 1

I understood that multiple machines (independent computers) are involved in your system, right? Then, first of all, you need to make sure that clocks of the various machines sending events are synchronized to a certain degree. However, since there will always be a time sync error you have to make up your mind about what you mean with "all the events that has exactly the same timestamp". From my point of view, you need to define an arbitrary time delta value. You then simply define all events with timestamps within this time delta to have happened "at the same time".

Secondly, if I understood right, you want "producers" to send their events to a "collector" via TCP. This raises the question of whether you already have a certain communication protocol in place on top of which you could simply add the communication of timestamped events, or if you need to establish an independent communication layer (a distinct TCP socket on the collector).

In any case, you might want to create a simple event class that stores the event message (the log message and any required data) as well as the creation timestamp. The producers create instances of this class. You can then use pickle in order to transmit a certain event through a byte stream (TCP connection) to the collector. The collector can unpickle and process it.

Be aware of the fact that time.time() resolution on Linux is higher than on Windows.

Answer 2

You could add PythonLoggingObserver besides your FileLogObserver. Then configure python logger to send messages to syslog. Syslog could be configured to send messages to central server. On central server you can save messages to log file and have another app reading log file with LogReader.

You'll need NTP to sync clocks also.