Did grammar behaviour break between 6.c and 6.d?

Question

I have a grammar that ws working fine in 6.c (2018.01) - I cannot get it to give stable output in 6.d (2019.03.01).

When I turn on Grammar::Tracer (nice!), the pattern seems to be consistent.

But with same input 'm/s', my output wanders randomly through a range of results for example... * m/s=m.True * m/s=True.m * m/s=s-1.True * m/s=m.s-1 (this is the one I want, don't care about the order)

I am suspicious of the unam => ｢s｣ ｢/｣ ｢m｣ content of the match object (see debug code) -- the Tracer shows only one match for unam in each branch.

All advice welcome!

#!/usr/bin/env perl6

sub get-dime( $dime-str ) { 
        use Grammar::Tracer;

        my $unit-names = '|m|s';

        grammar UnitGrammar {
            token TOP     { <dim> <divi> <den> }
            token divi    { \/ }
            token dim     { <unam> }
            token den     { <unam> }
            token unam    { <$unit-names> }
        }    
        class UnitActions {
            method TOP($/)     { make $/.values.[0].made~'.'~$/.values.[1].made }
            method dim($/)     { make ~$<unam> }
            method den($/)     { make ~$<unam>~'-1' }
            method divi($/)    { make True }
        }    

        my $match = UnitGrammar.parse($dime-str, :actions(UnitActions));

        #[[ verbose for debug (also uncomment #use Grammar::Tracer)
        say "$match=", $match.made;
        say "--------------------";
        say $match.values;
        say $match.values.[0].made;
        say $match.values.[1].made;
        #]]  

        return $match.made if $match.so;
}
say $*PERL.compiler.version;
say get-dime( 'm/s' );

ban

Answer 1

@Håkon named the game. You teed off to land on the courseway. I got us near the green but in the rough. @ugexe spotted the ball. @Håkon skipped this hole, moved to the next tee and holed in one. So I guess I'm left to write the post-game wrap up.

---

It's not 6.c vs 6.d.

The underlying issue is that hashes are specified to have their key/value pairs listed in random order. In earlier compilers the hash list implementation returned key/value pairs in a fixed order. Then samcv implemented hash randomization for MoarVM last year. It breaks code that incorrectly relies on hash order.

So the following code -- on Rakudo/MoarVM since 2018.05 -- randomly displays either (a b) or (b a):

say .values given { apple => 'a', bananas => 'b' }

Your code was calling .values on a Match object.

Calling that method will usually boil down to something similar in effect to:

say .values given ( apple => 'a', bananas => 'b', 0, 1, 2 ) .Capture

which will display a list starting with 0 1 2 (they're positionals, so they display first, in their correct order) followed by either a b or b a (they're named so they display last, in random order).

This in turn is because a Match is a sub-class of Capture and inherits its .values method from that class. As its doc says:

Returns a Seq containing all positional values followed by all named argument values.

In your code there were no positional captures. So all you got to see was the values of named captures. But of course they're in a list, so appear under indices .[0], .[1], etc. which presumably tricked you into thinking they were ordered (in combination with the fact that, in earlier Rakudos, their supposedly random order was in reality fixed).

.caps improves on this because it orders its values according to their match positions in the input. This will ensure stable ordering of a list of the values of named captures -- provided none of the captures share a matching position:

say .caps given 'foo' ~~ / $<alias>=<ident> /

will randomly display either (alias => ｢foo｣ ident => ｢foo｣) or (ident => ｢foo｣ alias => ｢foo｣).

My first answer

For the record, here was the first version of this answer:

Here it is golf'd about as far down as I think it can go:

grammar {
  token TOP { <one> <two> }
  token one { 1 }
  token two { 2 }
}.parse('12').values.say;

In the v2018.04 that's the current version running on tio.run this reliably displays (｢1｣ ｢2｣). In the v2018.12 that's the current version running on glot.io this produces either of (｢1｣ ｢2｣) or (｢2｣ ｢1｣), randomly varying between the two.

I don't know if this is a bug. Investigation continues but I thought I'd post this as an initial answer.

Answer 2

I am not sure what order $/.values is supposed to return (I get the randomized order as you describe), but instead of using $/.values:

method TOP($/)     {
     make $<divi> ?? $/.values.[0].made~'.'~$/.values.[1].made
                  !! $/.values.[0].made
}

you can use try $/.caps:

method TOP($/)     {
    make $<divi> ?? $/.caps.[0].value.made~'.'~$/.caps.[2].value.made
                 !! $/.caps.[0].value.made
}

at least this works for me.