#include<stdio.h>
int main()
{
int a,b;
a=a+b;
printf("%d",a);
return 0;
}
what should be the output if this code is passed through a lexer
1.1 Creating a Lexer. Produces a function that takes an input-port, matches the re's against the buffer, and returns the result of executing the corresponding action-expr.
Lexer is used to pre-process the source code, so as to reduce the complexity of parser. Lexer is also a kind of compiler which consumes source code and output token stream. lookahead(k) is used to fully determine the meaning of current character/token.
They are called scannerless parsers. A lexer and a parser work in sequence: the lexer scans the input and produces the matching tokens, the parser then scans the tokens and produces the parsing result.
The lexer is contained in the file lex.cc . It is a hand-coded lexer, and not implemented as a state machine. It can understand C, C++ and Objective-C source code, and has been extended to allow reasonably successful preprocessing of assembly language.
the lexer just tokenizes the stream to turn a stream of characters into a stream of tokens (that will be parsed with a parser later to obtain a full syntax tree). For your example you would obtain something like:
#include <stdio.h> (this is handled by preprocessor, not by lexer so it wouldn't exist)
int KEYWORD
main IDENTIFIER
( LPAR
) RPAR
{ LBRACE
int KEYWORD
a IDENT
, COMMA
b IDENT
; SEMICOL
a IDENT
= ASSIGN
a IDENT
+ PLUS
b IDENT
; SEMICOL
printf IDENT
( LPAR
"%d" STRING
, COMMA
a IDENT
) RPAR
; SEMICOL
return RETURN_KEYWORD
0 INTEGER
; SEMICOL
} RBRACE
Of course a lexer by itself can't do much, it can just split the source into smallest elements possible, checking for syntax errors (like misspelled keywords). You will need something that will combine them to give them a semantic meaning.
Just a side note: some lexers like to group similar kinds of tokens in just one (for example a KEYWORD
token that contains all keywords) using a parameter associated with it, while others have a different token for every one like RETURN_KEYWORK
, IF_KEYWORD
and so on..
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With