Redefining ws in a grammar

Question

According to the documentation, you can redefine the ws token in a grammar, this token is called automatically in some cases, such as this:

grammar Numbers { rule TOP { \d \d } }; 
my $result = Numbers.parse("3 \n 3");
say $result.perl
# OUTPUT: «Match.new(pos => 5, made => Any, from => 0, hash => Map.new(()), orig => "3 \n 3", list => ())␤»  

One of the advantages of redefining ws might be that it will not be thrown away. OK, I'll buy that and use for ws the exact same definition that is used internally:

grammar Numbers { rule TOP { \d \d }; regex ws { <!ww> \s* } };
my $result = Numbers.parse("3 \n 3");
say $result<ws> # OUTPUT: «Nil␤» 

Matching works, but $result is still dropped (redefining this to another token that does not use the default ws will work). So is ws dropped always?

Update This is probably related to this Rakudo bug

Answer 1

Whitespace not being captured is nothing to do with the definition of ws, but rather to do with the way sigspace ("significant whitespace") works.

Sigspace, enabled by the :s modifier and on by default in a rule, inserts <.ws> according to its rules (which can be summarized as "after an atom"). This is a non-capturing call to the ws rule. Redefining ws has no impact on this, since it's a property of the rule calling ws, not of ws itself.

Indeed, if we write an explicit call to the default <ws>:

say "1 2" ~~ /\d <ws> \d/

The it will capture, the output of the above program being:

｢1 2｣
 ws => ｢ ｣