perl sort question

Question

I have some huge log files I need to sort. All entries have a 32 bit hex number which is the sort key I want to use. some entries are one liners like

bla bla bla  0x97860afa bla bla  

others are a bit more complex, start with the same type of line above and expand to a block of lines marked by curly brackets like the example below. In this case the entire block has to move to the position defined by the hex nbr. Block example-

 bla bla bla  0x97860afc bla bla  
     bla bla {  
         blabla  
            bla bla {  
                bla     
            }  
        }  

I can probably figure it out but maybe there is a simple perl or awk solution that will save me 1/2 day.

---

Transferring comments from OP:

Indentation can be space or tab, I can enhance that on any proposed solution, I think that Brian summarizes well: Specifically, do you want to sort "items" which are defined as a chunk of text that starts with a line containing a "0xNNNNNNNN", and contains everything up to (but not including) the next line which contains a "0xNNNNNNNN" (where the N's change, of course). No lines interspersed.

Answer 1

Something like this might work (Not tested):

my $line;
my $lastkey;
my %data;
while($line = <>) {
    chomp $line;
    if ($line =~ /\b(0x\p{AHex}{8})\b/) {
        # Begin a new entry
        my $unique_key = $1 . $.; # cred to [Brian Gerard][1] for uniqueness
        $data{$1} = $line;
        $lastkey = $unique_key;
    } else {
        # Continue an old entry
        $data{$lastkey} .= $line;
    }
}

print $data{$_}, "\n" for (sort { $a <=> $b } keys %data);

The problem is that you said "huge" log files, so storing the file in memory will probably be inefficient. However, if you want to sort it, I suspect you're going to need to do that.

If storing in memory is not an option, you can always just print the data to a file instead, with a format that will allow you to sort it by some other means.

Answer 2

• For Huge data files, I'd recommend Sort::External.
• It doesn't look like you need to parse the brackets, if the indentation does the job. Then you have to do it on "breaks", or when the indentation level 0, then you process the last record gathered, so you always look ahead one line.

So:

sub to_sort_form {
    my $buffer = $_[0];
    my ( $id ) = $buffer =~ m/(0x\p{AHex}{8})/; # grab the first candidate
    return "$id-:-$buffer";
    $_[0] = '';
}

sub to_source { 
    my $val = shift;
    my ( $record ) = $val =~ m/-:-(.*)/;
    $record =~ s/\$--\^/\n/g;
    return $record;
}

my $sortex = Sort::External->new(
      mem_threshold   => 1024**2 * 16     # default: 1024**2 * 8 (8 MiB)
    , cache_size      => 100_000          # default: undef (disabled) 
    , sortsub         => sub { $Sort::External::a cmp $Sort::External::b }
    , working_dir     => $temp_directory  # default: see below
);

my $id;
my $buffer = <>;
chomp $buffer;
while ( <> ) { 
    my ( $indent ) = m/^(\s*)\S/;
    unless ( length $indent ) {
        $sortex->feed( to_sort_form( $buffer ));
    }
    chomp;
    $buffer .= $_ . '$--^';
}
$sortex->feed( to_sort_form( $buffer ));
$sortex->finish;

while ( defined( $_ = $sortex->fetch ) ) {
    print to_source( $_ );
}

Assumptions:

• The string '$--^' does not appear in the data on its own.
• That you're not alarmed about two 8-hex-digit strings in one record.