Need to compare times in Perl; how to deal with timezone?

Question

I have log files that have timestamps like:

Fri Nov 30 10:19:35:152.92 PST 2012
Fri Nov 30 10:19:35:228.8 PST 2012
or even:
Thu Nov 29 14:20:58:3.44 PST 2012
Fri Nov 30 10:27:50:742 PST 2012

I am very new to Perl, but it is what everyone here uses, so I'm trying to learn it fast (just started this job). I need to be able to compare the timestamps (I'm merging logfiles whose times may overlap and need all the timestamps consecutive in the resultant file). Here is the subroutine where I extract and format the time into a string I can compare:

my %months = ( 'Jan'=>1, 'Feb'=>2, 'Mar'=>3, 'Apr'=>4, 'May'=>5, 'Jun'=>6, 'Jul'=>7,
'Aug'=>8, 'Sep'=>9, 'Oct'=>10, 'Nov'=>11, 'Dec'=>12);

sub to_comparable {
    my $date = shift;
    my ($mmm, $d, $H, $M, $S, $mils, $fra, $tz, $Y) = $date =~ 
        m{^<\w{3} (\w{3}) (\d{1,2}) (\d{1,2}):(\d{1,2}):(\d{1,2}):(\d{1,3})[.]{0,1}(\d{0,2}) (\w{3}) (\d{4})>}
            or return undef;
    if ($mils eq "") { $mils = 0; }
    if ($fra eq "") { $fra = 0; }
    my $m = $months{$mmm};
    return sprintf('%04d%02d%02d%02d%02d%02d%03d%02d',$Y,$m,$d,$H,$M,$S,$mils,$fra);
}

This works fine as long as the timestamps all come from the same timezone. However, I want to make sure that they work OK for logs that overlap the change from standard time and daylight savings time (or if I get logs from other timezones). I thought perhaps the DateTime package would do the trick, but I'm confused as to how to use the timezone to get comparable times. I can create a date/time object except for what to use for the timezone. In my testing, I added the following after mapping the month:

    my $ns = sprintf('%03d.%02d',$mils,$fra);
    $ns *= 1000;

    my $dt = DateTime->new(
      year       => $Y,
      month      => $m,
      day        => $d,
      hour       => $H,
      minute     => $M,
      second     => $S,
      nanosecond => $ns,
      time_zone  => "$tz",
  );

This results in the error "Invalid offset: PST". I found the note:"It is strongly recommended that you do not rely on these names for anything other than display. These names are not official, and many of them are simply the invention of the Olson database maintainers. Moreover, these names are not unique. For example, there is an "EST" at both -0500 and +1000/+1100." and elsewhere: "The short names for time zones are not unique, and so any attempt to determine the actual time zone from such a name involves guessing. Use the long names instead."

I have no control over the timezone display I am given, so I don't know what to do now. If I use "PST8PDT" or "America/Los_Angeles", how do I indicate whether the given time is standard or daylight savings? And is there a conversion for the timezones in the US to timezones that DateTime will accept? Can someone please help me figure this out? My seemingly trivial project of merging log files is taking forever and my boss thinks I'm an idiot. :-(

Answer 1

It's a pity so many log files use such a poor choice of timestamp format. I recommend RFC 3339 (which is also one of the ISO 8601 formats).

Anyway, on to the question.

Create a mapping to translate your source system's identifiers definitions into the standard names or offsets accepted by the time_zone argument of the constructor.

my %time_zones = (
   EST => '-0500',
   PST => '-0800',
   PDT => '-0700',
   ...
);

Then just pass the offset using the time_zone argument.

$ perl -MDateTime -E'say
   DateTime->new(
      year => 2012, month => 11, day => 4,
      hour => 1, minute => 16, second => 0,
      time_zone => "-0800",
   )->epoch;
'
1352020560

$ perl -MDateTime -E'say
   DateTime->new(
      year => 2012, month => 11, day => 4,
      hour => 1, minute => 16, second => 0,
      time_zone => "-0700",
   )->epoch;
'
1352016960

$ perl -E'say 1352020560 - 1352016960'
3600