How to do python-like indentation with flex/bison

Question

I want my language to have two features that make Python such a nicely formatted language:

• One statement per line
• Blocks begin with another indentation level and go on until that's ended

Can anyone give me a detailed hint on how to achieve that with flex/bison-like tools? Such a block feature forces the user to write readable code.

Answer 1

You could try to track the indentation level in the lexer, and add pseudo-tokens for indent and unindent. You will need to keep a stack of already seen indentation levels, and need to care about empty/comment-only lines differently. But I fear that at the end the lexer will become an unmaintainable mess and also you have some parse-specific state (the indentation stack) in your lexer.

Answer 2

Matt Might wrote an article on standalone parsers, with a way of handling significant whitespace using "unput":

http://matt.might.net/articles/standalone-lexers-with-lex/

(The example is half-way down the page.)