Using grep and awk together

Question

I have a file (A.txt) with 4 columns of numbers and another file with 3 columns of numbers (B.txt). I need to solve the following problems:

1. Find all lines in A.txt whose 3rd column has a number that appears any where in the 3rd column of B.txt.

2. Assume that I have many files like A.txt in a directory and I need to run this for every file in that directory.

How do I do this?

Answer 1

You should never see someone using grep and awk together because whatever grep can do, you can also do in awk:

Grep and Awk

grep "foo" file.txt | awk '{print $1}'

Using Only Awk:

awk '/foo/ {print $1}' file.txt

I had to get that off my chest. Now to your problem...

Awk is a programming language that assumes a single loop through all the lines in a set of files. And, you don't want to do this. Instead, you want to treat B.txt as a special file and loop though your other files. That normally calls for something like Python or Perl. (Older versions of BASH didn't handle hashed key arrays, so these versions of BASH won't work.) However, slitvinov looks like he found an answer.

Here's a Perl solution anyway:

use strict;
use warnings;
use feature qw(say);
use autodie;

my $b_file = shift;
open my $b_fh, "<", $b_file;

#
# This tracks the values in "B"
#
my %valid_lines;
while ( my $line = <$b_file> ) {
    chomp $line;
    my @array = split /\s+/, $line;
    $valid_lines{$array[2]} = 1;   #Third column
}
close $b_file;

#
# This handles the rest of the files
#
while ( my $line = <> ) {  # The rest of the files
   chomp $line;
   my @array = split /\s+/, $line;
   next unless exists $valid_lines{$array[2]};  # Next unless field #3 was in b.txt too
   say $line;
}

Answer 2

Here is an example. Create the following files and run

awk -f c.awk B.txt A*.txt 

c.awk

FNR==NR {
    s[$3]
    next
}

$3 in s {
    print FILENAME, $0
}

A1.txt

1 2 3
1 2 6
1 2 5

A2.txt

1 2 3
1 2 6
1 2 5

B.txt

1 2 3
1 2 5
2 1 8

The output should be:

A1.txt 1 2 3
A1.txt 1 2 5
A2.txt 1 2 3
A2.txt 1 2 5