Antlr error : the following token definition can never be matched because prior tokens match the same input

Question

I am writhing a simple language with antlr, I defined a Lexer grammar in AntlrWorks, but when I want to generate the java code, it gives me the error:

Antlr error : the following token definition can never be matched because prior tokens match the same input: FLOAT_OR_INT, OPEN_PAR, CLOSE_PAR, .... (almost for all the rules!)

I am new to antlr, I assume it is because of the order of rule locations, but I don't know how should they have to be, what is my mistake?

here is the grammar:

lexer grammar OurCompiler;
options
{
    k=5;

} 


ID                      : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*
                        ;



protected
INT                     : ('0'..'9')+
                        ;

protected
FLOAT                   : INT '.' INT
                        ;


FLOAT_OR_INT            : ( INT '.' ) => FLOAT { $setType(FLOAT); }
                        | INT { $setType(INT); }
                        ;       


OPENPAR_OR_OUTPUT_OPERATOR  :   '(' { $setType(OPEN_PAR); } | '(' '(' { $setType(OUTPUT_OPERATOR); }
                            ;

CLOSEPAR_OR_INPUT_OPERATOR  :   ')' { $setType(CLOSE_PAR); } | ')' ')' { $setType(INPUT_OPERATOR); }
                            ;

protected   
OPEN_PAR                :  '('  ;

protected
CLOSE_PAR               :  ')'  ;

protected
INPUT_OPERATOR          :   ')' ')' ;

protected
OUTPUT_OPERATOR         :   '(' '(' ;



BOOLEAN                 :   't' 'r' 'u' 'e' | 'f' 'a' 'l' 's' 'e'   ;    

LOWER                   :   '<'     ;

LOWER_EQUAL             :   LOWER '='   ;

UPPER                   :  '>'  ;

UPPER_EQUAL             :    UPPER '='  ;

ASSIGN                  : '='  ;

EQUAL                   :  '=' '='  ;

NOT                     :   '!'  ;

NOT_EQUAL               :  NOT '='  ;

ADD                     :   '+'  ;

ADD_TO_PREVIOUS         :  ADD '='  ;

INCREMENT               :   ADD ADD  ;

MINUS                   : '-'  ;

MINUS_FROM_PREVIOUS     :  MINUS '='  ;

DECREMENT               :  MINUS MINUS  ;

MULTIPLY                :  '*'  ;

MULTIPLY_TO_PREVIOUS    :  MULTIPLY '='  ;

DIVIDE                  :  '/'  ;

DIVIDE_FROM_PREVIOUS    :  DIVIDE '='  ;

MODE                    :  '%'  ;

OPEN_BRAKET             :  '['  ;

CLOSE_BRAKET            :  ']'  ;

OPEN_BRACE              :  '{'  ;

CLOSE_BRACE             :  '}'  ;

COLON                   : ':'   ;

SEMICOLON               :  ';'  ;

COMMA                   :  ','  ;


SINGLE_LINE_COMMENT     : 
                        '#' '#' ( ~ ('\n'|'\r') )*  ( '\n' | '\r' ('\n')? )? { $setType(Token.SKIP); newline(); }
                        ; 


MULTIPLE_LINE_COMMENT   :   '#' ( options {greedy=false;} : . )* '#' { $setType(Token.SKIP); }
                        ;



WS                      : 
                        ( ' '
                        | '\t'
                        | '\r'    { newline(); }
                        | '\n'    { newline(); }
                        )
                        { $setType(Token.SKIP); } 
                        ;


protected
ESC_SEQ                 :   '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
                    ;

STRING                  :  
                        '"' ( ESC_SEQ | ~('\\'|'"') )* '"'
                        ;

CHAR                    :
                        '\'' ( ESC_SEQ | ~('\''|'\\') ) '\''
                        ;





INT_KEYWORD             :   'i' 'n' 't'         ;

FLOAT_KEYWORD           :   'f' 'l' 'o' 'a' 't'     ;

CHAR_KEYWORD            :   'c' 'h' 'a' 'r'         ;

STRING_KEYWORD          :   's' 't' 'r' 'i' 'n' 'g'     ;

BOOLEAN_KEYWORD         :   'b' 'o' 'o' 'l' 'e' 'a' 'n'     ;

INPUT_KEYWORD           :   'i' 'n' ID  { $setType(ID); }
                        |   'i' 'n'     
                        ;

OUTPUT_KEYWORD          :   'o' 'u' 't' ID  { $setType(ID); }
                        |   'o' 'u' 't'     ;

IF_KEYWORD              :   'i' 'f'             ;

FOR_KEYWORD             :   'f' 'o' 'r'         ;

SWITCH_KEYWORD          :   's' 'w' 'i' 't' 'c' 'h'     ;

CASE_KEYWORD            :   'c' 'a' 's' 'e'         ;

BREAK_KEYWORD           :   'b' 'r' 'e' 'a' 'k'     ;

DEFAULT_KEYWORD         :   'd' 'e' 'f' 'a' 'u' 'l' 't' ;

WHILE_KEYWORD           :   'w' 'h' 'i' 'l' 'e'     ;

ELSE_KEYWORD            :   'e' 'l' 's' 'e'         ;

ELSEIF_KEYWORD          :   'e' 'l' 's' 'e' 'i' 'f'     ;

AND_KEYWORD             :   'a' 'n' 'd'         ;

OR_KEYWORD              :   'o' 'r'             ;

NOT_KEYWORD             :   'n' 'o' 't'         ;

CONSTANT_KEYWORD        :   'c' 'o' 'n' 's' 't' 'a' 'n' 't' ;

Answer 1

I have 7 remarks about your grammar after glancing over it:

1

k=? denotes the look-ahead for parser rules and since yours is a lexer grammar, remove it;

2

Although not wrong, BOOLEAN_KEYWORD : 'b' 'o' 'o' 'l' 'e' 'a' 'n'; is rather verbose. Do BOOLEAN_KEYWORD : 'boolean'; instead.

3

The keyword protected has changed in ANTLR 3 to fragment. But you're doing odd things. Take the following rules:

fragment
INT 
 : ('0'..'9')+
 ;

fragment
FLOAT 
 : INT '.' INT
 ;

FLOAT_OR_INT 
 : ( INT '.' ) => FLOAT { $setType(FLOAT); }
 | INT { $setType(INT); }
 ;

You create two fragments, and then have FLOAT_OR_INT check through a predicate if it "sees" an INT followed by a '.' and then change it into a FLOAT. The following does the same and is far more readable/better/preferred:

FLOAT 
 : DIGIT+ '.' DIGIT+
 ;

INT 
 : DIGIT+
 ;

fragment DIGIT 
 : '0'..'9'
 ;

4

.* is ungreedy by default, so change:

'#' ( options {greedy=false;} : . )* '#'

into

'#' .* '#'

or even better:

'#' ~'#'+ '#'

5

The rule:

OPENPAR_OR_OUTPUT_OPERATOR
 : '('     { $setType(OPEN_PAR); } 
 | '(' '(' { $setType(OUTPUT_OPERATOR); }
 ;

should simply be:

OUTPUT_OPERATOR
 : '(('
 ;

OPEN_PAR
 : '('
 ;

6

ANTLR's lexer tries to match as much characters as possible. Whenever two rules match the same amount of characters, the rule defined firs will "win". That is why you should define all your *_KEYWORD rules before the ID rule.

7

Lastly, you don't need to check if "in" or "out" is followed by an ID (and then change the type of the token). Whenever the lexer "sees" input like "inside", it will always create a single ID token, and not an INPUT_KEYWORD followed by an ID, since the lexer matches as much as possible (see remark #6).

---

It appears you're trying to learn ANTLR by trial and error, or are using out-dated documentation. This is not the way to learn ANTLR. Try to get a hold of Parr's The Definitive ANTLR Reference to learn it properly.

Good luck!

EDIT

Well, in case you don't manage to get it working, here's a working version of your grammar:

lexer grammar OurCompiler; // A bit of an odd name for a lexer...

K_INT      : 'int';
K_FLOAT    : 'float';
K_CHAR     : 'char';
K_STRING   : 'string';
K_BOOLEAN  : 'boolean';
K_INPUT    : 'in';
K_OUTPUT   : 'out';
K_IF       : 'if';
K_FOR      : 'for';
K_SWITCH   : 'switch';
K_CASE     : 'case';
K_BREAK    : 'break';
K_DEFAULT  : 'default';
K_WHILE    : 'while';
K_ELSE     : 'else';
K_ELSEIF   : 'elseif';
K_AND      : 'and';
K_OR       : 'or';
K_NOT      : 'not';
K_CONSTANT : 'constant';

BOOLEAN : 'true' | 'false';
FLOAT   : DIGIT+ '.' DIGIT+; 
INT     : DIGIT+;
STRING  : '"'  ( ESC_SEQ | ~('\\'|'"') )* '"';
CHAR    : '\'' ( ESC_SEQ | ~('\''|'\\') ) '\'';

ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*;

INPUT_OPERATOR  : '))';
OUTPUT_OPERATOR : '((';
OPEN_PAR        : '(';
CLOSE_PAR       : ')';

LOWER                : '<';
LOWER_EQUAL          : '<=';
UPPER                : '>';
UPPER_EQUAL          : '>=';
ASSIGN               : '=';
EQUAL                : '==';
NOT                  : '!';
NOT_EQUAL            : '!=';
ADD                  : '+';
ADD_TO_PREVIOUS      : '+=';
INCREMENT            : '++';
MINUS                : '-';
MINUS_FROM_PREVIOUS  : '-=';
DECREMENT            : '--';
MULTIPLY             : '*';
MULTIPLY_TO_PREVIOUS : '*=';
DIVIDE               : '/';
DIVIDE_FROM_PREVIOUS : '/=';
MODE                 : '%';
OPEN_BRAKET          : '[';
CLOSE_BRAKET         : ']';
OPEN_BRACE           : '{';
CLOSE_BRACE          : '}';
COLON                : ':';
SEMICOLON            : ';';
COMMA                : ',';

SINGLE_LINE_COMMENT   : '##' ~('\r' | '\n')*        {skip();}; 
MULTIPLE_LINE_COMMENT : '#' ~'#'+ '#'               {skip();};
WS                    : ( ' ' | '\t' | '\r' | '\n') {skip();};

fragment ESC_SEQ : '\\' ('b' | 't' | 'n' | 'f' | 'r' | '\"' | '\'' | '\\');
fragment DIGIT   : '0'..'9';