Including Hashes within Hashes in Perl

Question

G'Day,

I'm currently working on creating big hashes from a lot of smaller hashes. Let's say that these smaller hashes are defined in a file each, and then can be included by the bigger hash.

For eg, let's look at some small hashes

File personcontact.pl:

   return {
            \'firstname\' => {
                \'__type\' => \'String\'
            },
        \'lastname\' =>  {
            \'__type\' => \'String\'
            },
        %{include("/tmp/address.pl")}
    }

File address.pl:

return {
        \'address\' => {
        \'street\' => {
            \'__type\' => \'String\'
            },
        \'unit\' => {
            \'__type\' => \'String\',
            \'__validation_function\' => {
                \'is_a_number\' => \'\'
            },
            \'__schema_constraints\' => {
                \'is_not_null\' => \'\'
            }
        },
        \'suburb\' => {
            \'__type\' => \'String\'
        },
        \'__type\' => \'ARRAY\'
        }
    } 

And I've got a considerable number of these...

The way I'm trying to re-create the hash is using the include subroutine, which looks like this:

 sub include {
my ($filename) = @_;
my $file; 
open(my $fh, "<", $filename) or die ("FILEOPEN: $!");
while(my $line = <$fh>) { $file .= $line; }
my $result = eval $file;
die("EVAL: $@") if $@;
close($fh) or die("FILECLOSE: $!");
return $result;
 }

I know I must be doing something wrong, but I'm not sure what. I keep on getting errors like Useless use of a variable in void context at (eval 11) line 4, <SCHEMAFILE> line 6 or Odd number of elements in anonymous hash at (eval 11) line 5, <SCHEMAFILE> line 6. I'm not sure how to go about finding (eval 11) line 4-3, line 6 though. Any suggestions on use of Perl debuggers or any pointers on where I might be going wrong will be much appreciated.

Thanks!

Answer 1

Welcome to Perl. I hope you have a good time learning and using it.

On to business, where to start? I've got a lot to say here.

First off, it's unnecessarily risky to load data by evaluating files. If you just want to serialize data try JSON::XS or YAML, or even Storable. If you want a config file, there are many, many modules on CPAN that help with this task. Check out Config::Any.

If you want to create data structures to load via eval (which is not really a good idea), Data::Dumper generates perl code needed to create any data structures you feed to it. The main reason I mention it is that it is far more useful as a debugging aid than a serializer.

Now that that is taken care of, if you want to load a file and evaluate it (again, not the best idea in nearly every case), you should be looking at do or require.

my $stuff = do 'address.pl';

But don't do that. String eval is a tool that is generally best left unused. 99% of the time, if you are planning on using string eval, stop and think of another way to solve the problem. Do is an implicit eval, so it counts too.

Perl gives you lots of tools to do risky and powerful magic. A large part of becoming a skilled Perl programming lies in understanding what things are risky, why and when it makes sense to use them. Don't expect Perl to baby you with fences and gates to keep you safe. Seriously consider picking up a copy of Effective Perl Programming or Perl Best Practices. As a newbie, much will go over your head as you read the first time, but either book can be a great reference as you grow and learn.

Next topic, what in the world are you trying to achieve with all those escaped quotes? It makes my head hurt to look at that stuff! Perl has some very, very nice quoting operators that you can use to avoid ever having to mess around with escaping quotes in your literal strings.

The => or fat comma automatically quotes its left hand side (LHS) as if it is alphanumerics only. But putting all the quotes and escapes makes things really dodgy.

When you say \'address\' => {}, Perl sees this as the \, the "get reference" operator applied to a string literal. In this case, an unterminated string literal, because you never offer an unescaped ' after the first.

If your aim is to use 'address', quotes and all as your hash key, you can do this:

my %foo = ( "'address'" => 'blah' );

If you don't want the quotes, which seems a far more usual use case, simply do:

my %foo = ( address => 'blah' );

On to the error messages you were getting! Perl has some pretty nice error messages once you learn what they all mean. Until then, it can be a bit tough to understand their significance. Fortunately Perl ships with a script called splain: a handy dandy tool that will explain error messages in much greater detail. You can also use the diagnostics module to get the same, expanded error messages automatically.

Now, if I was writing this I'd do something along these lines:

gen_schema_files.pl - A file to write JSON schema files. You could hand edit your schemata if you want to. You may also want to configure the output to be prettier if you want to improve readability.

#!/usr/bin/perl

use JSON::XS;
use File::Spec;

use constant BASEDIR => '.';

# Key is the file name, value is the data to put into the file.
my %schemata = (
    'address.json' => {
        address => {
            street => { __type => 'String' },
            unit => {
                __type => 'String',
                __validation_function => { is_a_number => '' },
                __schema_constraints  => { is_not_null => ''  }
            },
            suburb => { __type => 'String' },
            __type => 'ARRAY'
        },
    },

    'person_contact.json' => {
         firstname => { __type => 'String' },
         lastname =>  { __type => 'String' },

         # Use a special key to indicate that additional files should be 
         # loaded into this hash.
         INCLUDE  => [qw( address.json )], 
     },

     # And so forth
);

for my $schema ( keys %schemata ) {
    my $path = File::Spec->catfile( BASEDIR, $schema );

    open my $fh, '>', $path
        or die "Error opening '$path' for writing - $!\n";

    print $fh encode_json $schemata{$schema};
}

load_schemas.pl - this is the code that loads the schemata and does stuff. Mine only gets loaded. I have no idea what you are doing with the data...

#!/usr/bin/perl
use strict;
use warnings;

use Data::Dumper;

use JSON::XS;
use File::Spec;

use constant BASEDIR => '.';


my $schema = load_schema( 'person_contact.json' );

print Dumper $schema;


sub load_schema {
    my $file = shift;

    my $path = File::Spec->catfile( BASEDIR, $file );

    open my $fh, '<', $path
        or die "Error opening file '$path' - $!\n";

    my $json = join '', <$fh>; # reads a list of lines and cats them into one string.
                               # One way to slurp out of many.

    my $schema = decode_json( $json );

    # Handle the inclusion stuff:

    if( exists $schema->{INCLUDE} ) {
        # Copy the files to load into an array.
        my @loadme = @{$schema->{INCLUDE}};
        # delete the magic special include key.
        delete $schema->{INCLUDE};

        # Load each file and copy it into the schema hash.
        for my $load ( @loadme ) {
            my $loaded = load_schema( $load );

            # This is a bit of weird syntax.
            # We are using a hash slice assignment to copy the loaded data into the existing hash.
            # keys and values are guaranteed to come out in the same (random) order as each other.
            # the @{$foo}{blahbhal} is how you dereference a hash reference as a slice.
            @{$schema}{keys %$loaded} = values %$loaded;
        }
    }

    return $schema;
}

I've glossed over a few things, but I've tried to leave comments with enough of the terms (vocabulary or even jargon, if you like) to allow you to do profitable searches.

The above code has a couple flaws. It does not do any checking for circular inclusions (it will run for a long time and eventually fill up memory and crash--not so good). The choice of magic key may not be good. And there are probably more I haven't even thought of yet.

Perldoc is an amazing resource, but there is so much there that it takes a while to learn to find things. Take a look at the Perl Data Structures Cookbook and the Arrays of Arrays tutorial. As a beginner, I found the Perl Functions by Category section of perlfunc to be incredibly helpful.

I think I'll stop, now that I've written more than enough text to blind the average person. I hope you find this dissertation helpful. Welcome, once again, and good evening (please adjust to whatever your local time is when you find this response).