How can I implement #include constructs with Flex and YACC?

Question

During parsing, if I encounter a include token I want to instruct YACC to open the file specified as input and to begin parsing this. Once this parsing is finished, I want to instruct YACC to return to the file and continue parsing directly after the include expression. I will restrict the include depth level to one.

Answer 1

The flex manual covers how to do this using yypush_buffer_state() and yypop_buffer_state().

Here is the section of the official manual on using multiple input buffers. There is some sample code.

Answer 2

It's normal to communicate between the lexical and syntactic phases of your processor.

So, recognize the syntax for an include directive in your parser (or, to make things easier, just recognize it in the lexer) and do the switching at the lexical level.

For example, here is a simple language that recognizes standard input lines containing ab or cd or .file. When it sees .someString it opens someString as an include file and then goes back to reading standard input.

%{
#include <stdio.h>
#include <stdlib.h>
void start_include(char *); int yylex(void); void yyerror(void *);
#define YYSTYPE char *
%}
%%
all:          all others | others;
others:       include_rule | rule_1 | rule_2 | 'Z' { YYACCEPT; };
include_rule: '.' '\n' { start_include($1); };
rule_1:       'a' 'b' '\n' { printf("random rule 1\n"); };
rule_2:       'c' 'd' '\n' { printf("random rule 2\n"); };
%%
FILE * f = NULL;
void start_include(char *s) {
        if ((f = fopen(s, "r")) == NULL)
                abort();
}
int yylex(void) {
        int c;
        static char s[100];
        if (f == NULL)
                f = stdin;
        c = getc(f);
        if (c == EOF) {
                f = stdin;
                c = getc(f);
        }
        if (c == '.') {
                scanf(" %s", s);
                yylval = s;
        } else if (c == EOF)
                return 'Z';
        return c;
}

And when we run it...

$ cat > toplevel
ab
.myinclude
ab
$ cat > myinclude
cd
cd
$ yacc ip.y && cc -Wall y.tab.c -ly && ./a.out < toplevel
random rule 1
random rule 2
random rule 2
random rule 1
$