AnTLR4: INDENT and DEDENT tokens

Question

I am implementing Python grammar in AnTLR4 but I am facing the same problem with INDENT and DEDENT discussed here: ANTLR4- dynamically inject token

The solution I am trying is to convert the solution by Ter that can be found here http://antlr3.org/grammar/1078018002577/python.tar.gz (override nextToken and insert imaginary tokens).

The problem is that this solution assumes that we have a lexer rule like:

LEADING_WS
    :   {getColumn()==1}?
        // match spaces or tabs, tracking indentation count
        (   ' '  { spaces++; }
        |   '\t' { spaces += 8; spaces -= (spaces % 8); }
        |   '\014' // formfeed is ok
        )+
        {
        }
        ...

but I keep getting an error because actions in lexer rule must be last element on single altermost alternative.

can anyone help me to find a solution?

Thanks a lot!!!

Answer 1

You need to move your calculation involving spaces to either the end of the LEADING_WS rule or your implementation of nextToken. At the end of LEADING_WS it could look like the following.

LEADING_WS
  : {getColumn()==1}?
    // match spaces or tabs, tracking indentation count
    [ \t]+
    {spaces = computeSpaces(_input.getText());}
    ;