Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Merge multiple log files into a single one based on timestamp

Tags:

shell

logging

I'm trying to find a solution to fastly merge 2 log files coming from 2 application servers. The log files is like this:

00:00:00,028 DEBUG [com.acme.productionservice...

I would like something that based on the time stamp print one line of the log file or another for example:

if file one have 2 lines:

00:00:00,028 DEBUG [com.acme.productionservice...
00:00:00,128 DEBUG [com.acme.productionservice...

and file two have this 3 lines:

00:00:00,045 DEBUG [com.acme.productionservice...
00:00:00,100 DEBUG [com.acme.productionservice...
00:00:00,150 DEBUG [com.acme.productionservice...

the output should be

00:00:00,028 DEBUG [com.acme.productionservice...   (file 1)
00:00:00,045 DEBUG [com.acme.productionservice...   (file 2)
00:00:00,100 DEBUG [com.acme.productionservice...   (file 2)
00:00:00,128 DEBUG [com.acme.productionservice...   (file 1)
00:00:00,150 DEBUG [com.acme.productionservice...   (file 2)

the only way I currently know is using cat file1 file | sort but this is very slow for gb of logs I need something like reading the 2 files and compare the timestamps and decide what to print.

like image 651
marcobazzani Avatar asked Sep 05 '25 11:09

marcobazzani


1 Answers

I ended up by using

sort -m 

I also used a trick to understand from which log file the log comes from with

for a in *.log ; do 
    awk  '$0=FILENAME" "$0' $a > $a.log
do
sort -m -k 2 *.log.log 
like image 149
marcobazzani Avatar answered Sep 08 '25 12:09

marcobazzani



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!