ANTLR4 PLSQL Grammar Python 3 Missing Function

Question

I'm using this grammar https://github.com/antlr/grammars-v4/tree/master/plsql for Antlr4 but I'm getting an error because its referring to a function that does not exist.

'NameError: name 'IsNewlineAtPos' is not defined '

I can see that someone has updated the csharp version two files (that are extra items? Base lexer and Parser? That include an update that refers to the function in the below code.

// https://docs.oracle.com/cd/E11882_01/server.112/e16604/ch_twelve034.htm#SQPUG054
REMARK_COMMENT:      'REM' {IsNewlineAtPos(-4)}? 'ARK'? (' ' ~('\r' | '\n')*)? NEWLINE_EOF -> channel(HIDDEN);

// https://docs.oracle.com/cd/E11882_01/server.112/e16604/ch_twelve032.htm#SQPUG052
PROMPT_MESSAGE:      'PRO' {IsNewlineAtPos(-4)}? 'MPT'? (' ' ~('\r' | '\n')*)? NEWLINE_EOF;

// TODO: should starts with newline
START_CMD
    //: 'STA' 'RT'? SPACE ~('\r' | '\n')* NEWLINE_EOF
    // https://docs.oracle.com/cd/B19306_01/server.102/b14357/ch12002.htm
    // https://docs.oracle.com/cd/B19306_01/server.102/b14357/ch12003.htm
    : '@' {IsNewlineAtPos(-2)}? '@'? ~('\r' | '\n')* NEWLINE_EOF
    ;

This is the update on git

https://github.com/antlr/grammars-v4/commit/94887a3f4c9040578ef01b561e1d5e0ec54cbe9b

I'm using Python to run this and would appreciate if someone could tell me why they 'extend' and what this function they create does. (and if they can show me how to convert to python 3 that would be awesome)

Answer 1

The {...}? parts are called semantic predicates and contain target specific code (Python in your case). So, that means IsNewlineAtPos(...) should be a function defined in a Python-flavored PlSqlBaseLexer, just like there is a PlSqlBaseLexer implementations for C# and Java:

# TODO import ANTLR's runtime classes here

class PlSqlBaseLexer(Lexer):

  __init__(self, ...):
    # TODO

  def IsNewlineAtPos(self, pos):
    # TODO

Normally, ANTLR creates a lexer from your grammar that extends ANTLR's own Lexer class, but since the PL-SQL grammar needs a bit more target specific code, the following is added to the grammar:

options {
    superClass=PlSqlBaseLexer;
}

which tells ANTLR that it needs to create a PlSqlBaseLexer that "sits" between ANTLR's own Lexer implementation and the PlSqlLexer. So instead of this inheritance tree:

Lexer
 '- PlSqlLexer

it now becomes:

Lexer
 '- PlSqlBaseLexer (including the `IsNewlineAtPos` function)
   '- PlSqlLexer

So there's really not more to it than copying what was done for the Java (or C#) version. Looking at the Java implementation:

protected boolean IsNewlineAtPos(int pos)
{
    int la = _input.LA(pos);
    return la == -1 || la == '\n';
}

combined with the predicate {IsNewlineAtPos(-4)}?, then the predicate will succeed when 4 characters back from the lexer's current character position (hence the minus 4) is -1 (nothing/EOF) or a newline character.

Also note the Usage, important note section in the README.