How do I implement include directives using boost::spirit::lex?

Question

I have a simple configuration file parser built from spirit::lex and spirit::qi. When the lexer reaches the pattern include "path" I want the text of the file to be included. As you may know, spirit::lexer::begin() starts the scanning process:

// Read file contents into a std::string
...

// _first and _last are const char*
_first = _contents.c_str();
_last  = &_first[_input.size()];

// _token is a lexer::iterator_type for the current token
_token = _lexer.begin(_first, _last);

My idea is to have a stack that stores lexer state represented as a struct:

struct LexerState
{
    const char* first;
    const char* last;
    std::string contents;
};

The lexer would be made to recognize the pattern for include "path" and in a semantic action extract the path to the include file. Then, the current lexer state is pushed on the stack, the file's contents are loaded into a string, and the new state is initialized as above using lexer::begin().

When the lexer finds the EOF character, the stack is popped and lexer::begin() is called using the previous lexer state variables.

Is it ok to repeatedly call lexer::begin() like this? How do I get lex::lexer to recognize the include "path" pattern and the EOF character without returning a token to the qi parser?

Finally, are there any alternative or better ways of accomplishing this?

Answer 1

Have a look at how Boost Wave does things:

The Wave C++ preprocessor library uses the Spirit parser construction library to implement a C++ lexer with ISO/ANSI Standards conformant preprocessing capabilities. It exposes an iterator interface, which returns the current preprocessed token from the input stream. This preprocessed token is generated on the fly while iterating over the preprocessor iterator sequence (in the terminology of the STL these iterators are forward iterators).

And regarding features:

The C++ preprocessor provides four separate facilities that you can use as you see fit:

• Inclusion of header files
• Macro expansion
• Conditional compilation
• Line control

Their Quick Start Sample shows how you'd use Boost Wave's lexer interface.