Sort a Huge file

Question

I want to sort a huge file of approx 20M rows:

• ascending on Team Name and
• then descending on highest score.

so I can get the highest scorers per team.

I want to be considerate of system's resources. So...

1. Is there a way to do this without putting all the data into a hash/array in Perl?
2. Can we do this using the Unix/Linux sort utility?

If so, can you please show how to do it?

My input-file will be about 20M rows in the following format

Chicago Bulls|Michael Jordan|38
LA Lakers|Kobe Bryant|32
Chicago Bulls|Steve Kerr|16
LA Lakers|Paul Gasol|20
LA Lakers|Shaquile ONeal|19
Chicago Bulls|Scottie Pippen|23
.
.
.

Answer 1

You don't need to sort.

 #!/usr/bin/perl
use warnings; use strict;
my %high_score;

while (<DATA>) {
    chomp;
    my ($team_name, $player, $score) = split(/\|/);
    for ($high_score{$team_name}{$player}) {
        $_ = $score
            unless $_ && $_ > $score
    }
}

for my $team_name (sort keys %high_score) {
    my %team_scores = %{ $high_score{$team_name} };
    my @top_players = sort { $team_scores{$b} <=>  $team_scores{$a} } (keys %team_scores);

    my $n = 0;
    for my $player (@top_players) {
        print "$team_name, $player high score: $team_scores{$player}\n";
        last if ++$n >= 2;
    }
}

__DATA__
Chicago Bulls|Michael Jordan|38
Chicago Bulls|Scottie Pippen|23
Chicago Bulls|Poor Joe|10
Chicago Bulls|Steve Kerr|16
LA Lakers|Kobe Bryant|32
LA Lakers|Paul Gasol|20
LA Lakers|Shaquile ONeal|19

Edits: (1) updated requirements (2) s/while/for/

Answer 2

I don't think you can tell sort to sort ascending in one column and descending in another. However, you can use two sorts in a pipeline using the -s option for stable sorting:

sort -t\| -rnk3 file.in | sort -st\| -k1