How do I get an Antlr Parser rule to read from both default AND hidden channel

Question

I use the normal whitespace separation into the hidden channel but I have one rule where I would like to include any whitespace for later processing but any example I have found requires some very strange manual coding.

Is there no easy option to read from multiple channels like the option to put the whitespace there from the beginning.

Ex. this is the WhiteSpace lexer rule

WS  :   ( ' '
        | '\t'
        | '\r'
        | '\n'
        ) {$channel=HIDDEN;}
    ;

And this is my rule where I would like to include whitespace

raw :   '{'? (~('{'))*;

Basically it's a catch all rule to capture any content that does not match other rules to be processed by another pattern and therefore I need the original stream.

I was hoping for a {$channel==DEFAULT || $channel==HIDDEN} syntax example but cannot find any.

My target will be C# but I can rewrite Java examples if required.

Answer 1

AFAIK, that is not possible. However, you could extend the UnbufferedTokenStream to change the channel during parsing. You can't use the CommonTokenStream since it buffers a variable amount of tokens (and there can be tokens in the buffer that are on the wrong channel!). Note that you need at least ANTLR 3.3: in previous versions the UnbufferedTokenStream wasn't included yet.

Let's say you want to parse (and display) either lower- or upper case letters. Upper case letters are put on the HIDDEN channel, so by deafult, only lower case letters will be parsed. However, when the parser stumbles upon a lower case "q", we want to change to the HIDDEN channel. Once parsing on the HIDDEN channel, we want the "Q" to bring us back to the DEFAULT_CHANNEL again.

So when parsing the source "aAbBcqCdDQeE", first "a", "b" and "c" are printed, then the channel is changed, then "C" and "D" get printed, then the channel is changed again, and finally "e" is printed to the console.

Here's an ANTLR grammar that does this:

ChannelDemo.g

grammar ChannelDemo;

@parser::members {
  private void handle(String letter) {
    if("Q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(Token.DEFAULT_CHANNEL);
    }
    else if("q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(HIDDEN);
    }
    else {
      System.out.println(letter);
    }
  }
}

parse
  :  any* EOF
  ;

any
  :  letter=(LOWER | UPPER) {handle($letter.getText());}
  ;

LOWER
  :  'a'..'z'
  ;

UPPER
  :  'A'..'Z' {$channel=HIDDEN;}
  ;

And here's the custom token stream class:

ChangeableChannelTokenStream.java

import org.antlr.runtime.*;

public class ChangeableChannelTokenStream extends UnbufferedTokenStream {

    public ChangeableChannelTokenStream(TokenSource source) {
        super(source);
    }

    public Token nextElement() {
        Token t = null;
        while(true) {
            t = super.tokenSource.nextToken();
            t.setTokenIndex(tokenIndex++);
            if(t.getChannel() == super.channel) break;
        }
        return t;
    }

    public void setChannel(int ch) {
        super.channel = ch;
    }
}

And a small Main class to test it all:

Main.java

import org.antlr.runtime.*;

public class Main {
    public static void main(String[] args) throws Exception {
        ANTLRStringStream in = new ANTLRStringStream("aAbBcqCdDQeE");
        ChannelDemoLexer lexer = new ChannelDemoLexer(in);
        ChangeableChannelTokenStream tokens = new ChangeableChannelTokenStream(lexer);
        ChannelDemoParser parser = new ChannelDemoParser(tokens);
        parser.parse();
    }
}

Finally, generate a lexer/parser (1), compile all source files (2) and run the Main class (3):

1

java -cp antlr-3.3.jar org.antlr.Tool ChannelDemo.g

2

javac -cp antlr-3.3.jar *.java

3 (*nix)

java -cp .:antlr-3.3.jar Main

3 (Windows)

java -cp .;antlr-3.3.jar Main

which will cause the following to be printed to the console:

a
b
c
C
D
e

EDIT

You can include the class in your grammar file like this:

grammar ChannelDemo;

@parser::members {
  private void handle(String letter) {
    if("Q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(Token.DEFAULT_CHANNEL);
    }
    else if("q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(HIDDEN);
    }
    else {
      System.out.println(letter);
    }
  }

  public static class ChangeableChannelTokenStream extends UnbufferedTokenStream {

    private boolean anyChannel;

    public ChangeableChannelTokenStream(TokenSource source) {
      super(source);
      anyChannel = false;
    }

    @Override
    public Token nextElement() {
      Token t = null;
      while(true) {
        t = super.tokenSource.nextToken();
        t.setTokenIndex(tokenIndex++);
        if(t.getChannel() == super.channel || anyChannel) break;
      }
      return t;
    }

    public void setAnyChannel(boolean enable) {
      anyChannel = enable;
    }

    public void setChannel(int ch) {
      super.channel = ch;
    }
  }
}

parse
  :  any* EOF
  ;

any
  :  letter=(LOWER | UPPER) {handle($letter.getText());}
  |  STAR                   {((ChangeableChannelTokenStream)input).setAnyChannel(true);}
  ;

STAR
  :  '*'
  ;

LOWER
  :  'a'..'z'
  ;

UPPER
  :  'A'..'Z' {$channel=HIDDEN;}
  ;

The parser that gets generated from the grammar above will enable reading from all channels when it encounters a "*". So when parsing "aAbB*cCdDeE":

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    ANTLRStringStream in = new ANTLRStringStream("aAbB*cCdDeE");
    ChannelDemoLexer lexer = new ChannelDemoLexer(in);
    ChannelDemoParser.ChangeableChannelTokenStream tokens =
        new ChannelDemoParser.ChangeableChannelTokenStream(lexer);
    ChannelDemoParser parser = new ChannelDemoParser(tokens);
    parser.parse();
  }
}

the following gets printed:

a
b
c
C
d
D
e
E