Incorrect Tokenization with Marpa

Question

I have a rather large Marpa grammar (for parsing XPath), and I ran into a problem with tokenization. I created a minimal breaking example below:

use strict;
use warnings;
use Marpa::R2;

my $grammar = Marpa::R2::Scanless::G->new(
    {
        source => \(<<'END_OF_SOURCE'),
            :default ::= action => ::array
            :start ::= Start

            Start  ::= Child DoubleColon Token

            DoubleColon ~ '::'
            Child ~ 'child'
            Token ~
                word
                | word ':' word
            word ~ [\w]+

END_OF_SOURCE
    }
);
my $reader = Marpa::R2::Scanless::R->new(
    {
        grammar => $grammar,
        trace_terminals => 1,
    }
);

my $input = 'child::book';
$reader->read(\$input);

This script prints the following:

Registering character U+0063 as symbol 10: [[\w]]
Registering character U+0063 as symbol 3: [[c]]
Registering character U+0068 as symbol 10: [[\w]]
Registering character U+0068 as symbol 4: [[h]]
Registering character U+0069 as symbol 10: [[\w]]
Registering character U+0069 as symbol 5: [[i]]
Registering character U+006c as symbol 10: [[\w]]
Registering character U+006c as symbol 6: [[l]]
Registering character U+0064 as symbol 10: [[\w]]
Registering character U+0064 as symbol 7: [[d]]
Registering character U+003a as symbol 1: [[\:]]
Rejected lexeme @0-5: Token; value="child"
Accepted lexeme @0-5: Child; value="child"
Registering character U+0062 as symbol 10: [[\w]]
Error in SLIF G1 read: No lexeme found at position 6
* String before error: child::
* The error  was at line 1, column 8, and at character 0x0062 'b', ...
* here: book

I want the input to be tokenized as [Child] [DoubleColon] [word]. As the terminal trace shows, only one colon character is read and processed. It seems that it tries to tokenize the beginning of the string as [word] [':'] [word] and fails partway through. The error will no longer be thrown if you remove line 10 of the grammar (| word ':' word).

I tried creating a priority for DoubleColon (:lexeme ~ <DoubleColon> priority > 1), but that didn't work. Can someone tell me what to do to make this grammar parse the input string correctly? It still needs to be able to parse child::ns:book, etc.

Answer 1

This appears to be a bug in the current release, 2.058, of Marpa::R2. My apologies and thank you for the careful writeup of the problem.

I have a fix, which passes the test suite, and I will get a new release out shortly.