extract last 10 minutes from logfile [duplicate]

Question

Trying to find a simple way for watching for recent events (from less than 10 minutes), I've tried this:

awk "/^$(date --date="-10 min" "+%b %_d %H:%M")/{p++} p" /root/test.txt

but it doesn't work as expected...

Log files are in form :

Dec 18 09:48:54 Blah
Dec 18 09:54:47 blah bla
Dec 18 09:55:33 sds
Dec 18 09:55:38 sds
Dec 18 09:57:58 sa
Dec 18 09:58:10 And so on...

Answer 1

Introduction

This answer is something long, because there is 3 different way on thinking: 1) perl quick or exact, 2) pure bash and 3) perl script in bash function.

That's a (common) job for perl!:

Simple and efficient:

perl -MDate::Parse -ne 'print if/^(.{15})\s/&&str2time($1)>time-600' /path/log

This version print last 10 minutes event, upto now, by using time function.

You could test this with:

sudo cat /var/log/syslog |
  perl -MDate::Parse -ne '
    print if /^(\S+\s+\d+\s+\d+:\d+:\d+)\s/ && str2time($1) > time-600'

Note that first representation use only firsts 15 chars from each lines, while second construct use more detailed regexp.

As a perl script: last10m.pl

#!/usr/bin/perl -wn

use strict;
use Date::Parse;
print if /^(\S+\s+\d+\s+\d+:\d+:\d+)\s/ && str2time($1) > time-600

Strictly: extract last 10 minutes from logfile

Meaning not relative to current time, but to last entry in logfile:

There is two way for retrieving end of period:

date -r logfile +%s
tail -n1 logfile | perl -MDate::Parse -nE 'say str2time($1) if /^(.{15})/'

Where logically, last modification time of the logfile must be the time of the last entry.

So the command could become:

perl -MDate::Parse -ne 'print if/^(.{15})\s/&&str2time($1)>'$(
    date -r logfile +%s)

or you could take the last entry as reference:

perl -MDate::Parse -E 'open IN,"<".$ARGV[0];seek IN,-200,2;while (<IN>) {
    $ref=str2time($1) if /^(\S+\s+\d+\s+\d+:\d+:\d+)/;};seek IN,0,0;
    while (<IN>) {print if /^(.{15})\s/&&str2time($1)>$ref-600}' logfile

Second version seem stronger, but access to file only once.

As a perl script, this could look like:

#!/usr/bin/perl -w

use strict;
use Date::Parse;
my $ref;                 # The only variable I will use in this.

open IN,"<".$ARGV[0];    # Open (READ) file submited as 1st argument
seek IN,-200,2;          # Jump to 200 character before end of logfile. (This
                         # could not suffice if log file hold very log lines! )
while (<IN>) {           # Until end of logfile...
    $ref=str2time($1) if /^(\S+\s+\d+\s+\d+:\d+:\d+)/;
};                       # store time into $ref variable.
seek IN,0,0;             # Jump back to the begin of file
while (<IN>) {
    print if /^(.{15})\s/&&str2time($1)>$ref-600;
}

But if you really wanna use bash

There is a very quick pure bash script:

Warning: This use recent bashisms, require $BASH_VERSION 4.2 or higher.

#!/bin/bash

declare -A month

for i in {1..12};do
    LANG=C printf -v var "%(%b)T" $(((i-1)*31*86400))
    month[$var]=$i
  done

printf -v now "%(%s)T" -1
printf -v ref "%(%m%d%H%M%S)T" $((now-600))

while read line;do
    printf -v crt "%02d%02d%02d%02d%02d" ${month[${line:0:3}]} \
        $((10#${line:4:2})) $((10#${line:7:2})) $((10#${line:10:2})) \
        $((10#${line:13:2}))
    # echo " $crt < $ref ??"   # Uncomment this line to print each test
    [ $crt -gt $ref ] && break
done
cat

Store this script and run:

cat >last10min.sh
chmod +x last10min.sh
sudo cat /var/log/syslog | ./last10min.sh

Strictly: extract last 10 minutes from logfile

Simply replace line 10, but you have to place filename in the script and not use it as a filter:

#!/bin/bash

declare -A month

for i in {1..12};do
    LANG=C printf -v var "%(%b)T" $(((i-1)*31*86400))
    month[$var]=$i
  done

read now < <(date -d "$(tail -n1 $1|head -c 15)" +%s)
printf -v ref "%(%m%d%H%M%S)T" $((now-600))

export -A month

{
    while read line;do
        printf -v crt "%02d%02d%02d%02d%02d" ${month[${line:0:3}]} \
            $((10#${line:4:2})) $((10#${line:7:2})) $((10#${line:10:2})) \
            $((10#${line:13:2}))
        [ $crt -gt $ref ] && break
    done
    cat
} <$1

A perl script into a bash function

As commented by ajcg, this could be nice to put efficient perl script into a bash function:

recentLog(){ 
    perl -MDate::Parse -ne '
        print if/^(.{'${3:-15}'})\s/ &&
            str2time($1)>time-'$((
                60*${2:-10}
            )) ${1:-/var/log/daemon.log}
}

Usage:

recentLog [filename] [minutes] [time sting length]

• filename of log file
• minutes max before now of lines to show
• time sting length from begin of lines (default 15).

Answer 2

You can match the date range using simple string comparison, for example:

d1=$(date --date="-10 min" "+%b %_d %H:%M")
d2=$(date "+%b %_d %H:%M")
while read line; do
    [[ $line > $d1 && $line < $d2 || $line =~ $d2 ]] && echo $line
done

For example if d1='Dec 18 10:19' and d2='Dec 18 10:27' then the output will be:

Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32

Or using awk if you wish:

awk -v d1="$d1" -v d2="$d2" '$0 > d1 && $0 < d2 || $0 ~ d2'