Datetime to epoch conversion

Question

I have a bash question (when using awk). I'm extracting every single instance of the first and fifth column in a textfile and piping it to a new file with the following code,

cut -f4 test170201.rawtxt | awk '/stream_0/ { print $1, $5  }' > testLogFile.txt 

This is part of the file (test170201.rawtxt) I'm extracting the data from, columns Timestamp and Loss,

Timestamp                 Stream     Status     Seq         Loss Bytes    Delay
17/02/01.10:58:25.212577  stream_0     OK      80281          0  1000     38473
17/02/01.10:58:25.213401  stream_0     OK      80282          0  1000     38472
17/02/01.10:58:25.215560  stream_0     OK      80283          0  1000     38473
17/02/01.10:58:25.216645  stream_0     OK      80284          0  1000     38472

This is the result I'm getting in testLogFile.txt

17/02/01.10:58:25.212577 0
17/02/01.10:58:25.213401 0
17/02/01.10:58:25.215560 0
17/02/01.10:58:25.216645 0

However, I want the Timestamp to be written in epoch in the file above. Is there an easy way of modifying the code I already have to do this?

Answer 1

Given:

$ cat file
Timestamp                 Stream     Status     Seq         Loss Bytes    Delay
17/02/01.10:58:25.212577  stream_0     OK      80281          0  1000     38473
17/02/01.10:58:25.213401  stream_0     OK      80282          0  1000     38472
17/02/01.10:58:25.215560  stream_0     OK      80283          0  1000     38473
17/02/01.10:58:25.216645  stream_0     OK      80284          0  1000     38472

You can write a POSIX Bash script to do what you are looking for:

while IFS= read -r line || [[ -n "$line" ]]; do
    if [[ "$line" =~ ^[[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{2} ]]
    then
        arr=($line)
        ts=${arr[0]}
        dec=${ts##*.}   # fractional seconds
        # GNU date may need different flags:
        epoch=$(date -j -f "%y/%m/%d.%H:%M:%S" "${ts%.*}" "+%s")
        printf "%s.%s\t%s\n" "$epoch" "$dec" "${arr[4]}"
    fi  
    done <file >out_file

$ cat out_file
1485975505.212577   0
1485975505.213401   0
1485975505.215560   0
1485975505.216645   0

For GNU date, try:

while IFS= read -r line || [[ -n "$line" ]]; do
    if [[ "$line" =~ ^[[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{2} ]]
    then
        arr=($line)
        ts="20${arr[0]}"
        d="${ts%%.*}"
        tmp="${ts%.*}"
        tm="${tmp#*.}"
        dec="${ts##*.}"    # fractional seconds
        epoch=$(date +"%s" --date="$d $tm" )
        printf "%s.%s\t%s\n" "$epoch" "$dec" "${arr[4]}"
    fi  
    done <file >out_file

For an GNU awk solution, you can do:

awk 'function epoch(s){ 
                        split(s, dt, /[/:. ]/)
                        s="20" dt[1] " " dt[2] " " dt[3] " " dt[4] " " dt[5] " " dt[6]
                        return mktime(s) "." dt[7]} 
     /^[0-9][0-9]/      { print epoch($1), $5 }'    file >out_file

If you don't want the fractional second included in the epoch, they are easily removed.

Answer 2

awk -F '[.[:blank:]]+' '
   # use separator for dot and space (to avoid trailing time info)

   {
   # for line other than header
   if( NR>1) {
      # time is set for format "YYYY MM DD HH MM SS [DST]"
      #  prepare with valuable info
      T = "20"$1 " " $2
      # use correct separator
      gsub( /[\/:]/, " ", T)
      # convert to epoch
      E = mktime( T)

      # print result, adding fractionnal as mentionned later
      printf("%d.%d %s\n", E, $3, $7)
      }
     else {
        # print header (line 1)
        print $1 " "$7
        }
    }
   ' test170201.rawtxt \
   > Redirected.file

• self commented, code is longer for understanding purpose
• use of gnu awk for the mktime function not available in posix or older version

Oneliner a bit optimized here after

awk -F '[.[:blank:]]+' '{if(NR>1){T="20"$1" "$2;gsub(/[\/:]/," ", T);$1=mktime(T)}print $1" "$7}' test170201.rawtxt