Read a large file and output sections matching multiple parameters

Question

I rarely have to deal with scripting, so I'm up against a lack of knowledge for this problem.

I have a file >500mb in text, which is nicely sectioned, but I know there are 5 to 10 "bad" sections inside. The data within the sections can be evaluated pretty easily by a human, I don't know how to do it in a program.

I pick up a known good value in #Field MyField - however if that value did not appear in #FIELD LOCATION, something went wrong.

An example of two sections within the file looks like this. The first is 'bad' and the second is 'good'.

#START Descriptor
#FIELD LOCATION="http://path.to/file/here&Value=FOO&OtherValue=BLAH"
#FIELD AnythingElse
#FIELD MyField="BAR"
#END
#START Descriptor
#FIELD LOCATION="http://path.to/file/here&Value=BAR&OtherValue=BLAH"
#FIELD AnythingElse
#FIELD MyField="BAR"
#END

1. Sections start and end logically, with #START and #END

2. If #FIELD LOCATION does not exist, go to next section

3. If #FIELD MyField="BAR" and #FIELD LOCATION does not contain BAR, print all lines from this section to a new file.

4. Note - Clarification of #FIELD MyField="BAR" - this is a check value I put in by grabbing other info about the data as this file is being built (in my case it is a language indicator, such as EN or DE. so it would literally be #FIELD MyField="EN" Any other value in this field would be ignored, this isn't a record that matches my criteria.

I believe this can be done in Awk or Perl, I can do very simple one-liners but this is beyond my skills.

Answer 1

You could do something like below. It's just a rough draft, but it will work with your sample data. Use the flip-flop operator to find the start and end of records. Use a hash to store the field values, and an array to store the record.

I am simply checking if the value is in the location string, you might wish to further narrow the check by making sure it is in the correct place, or the correct case.

use strict;
use warnings;

my @record;
my %f;
while(<DATA>) {
    if (/^#START / .. /^#END */) {
        if (/^#START /) {
            @record = (); # reset
            %f = ();
        }
        push @record, $_;
        if (/^#END */) { # check and print
            if ($f{'LOCATION'} !~ /$f{'MyField'}/) {
                print @record; 
            }
        } else {         # add fields to hash
            if (/^#FIELD (.+)/) {
                            # use split with limit of 2 fields
                my ($key, $val) = split /=/, $1, 2;
                next unless $val; # no empty values
                $val =~ s/^"|"$//g; # strip quotes
                $f{$key} = $val;
            }
        }
    }
}

__DATA__
#START Descriptor
#FIELD LOCATION="http://path.to/file/here&Value=FOO&OtherValue=BLAH"
#FIELD AnythingElse
#FIELD MyField="BAR"
#END
#START Descriptor
#FIELD LOCATION=http://path.to/file/here&Value=BAR&OtherValue=BLAH"
#FIELD AnythingElse
#FIELD MyField="BAR"
#END

Answer 2

One-liner:

perl -ne 'BEGIN { $/ = "#END\n" }' -e '/MyField="(.*?)"/; print if !/Value=$1/' <file >newfile

Sets the Input Record Separator to "#END\n" so perl reads the 'chunks' into $_ one at a time, then captures the value in MyField and prints the whole chunk if Value=$1 (that is, that capture after 'Value=') is not present.

You may of course make the regexes more specific if needed.