Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Boost Spirit X3 AST not working with semantic actions when using separate rule definition and instantiation

I am trying to use Boost Spirit X3 with semantic actions while parsing the structure to an AST. If I use a rule without separate definition and instantiation it works just fine, for example:

#include <vector>
#include <string>
#include <iostream>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/spirit/home/x3.hpp>

namespace ast 
{

struct ast_struct
{
  int number;
  std::vector<int> numbers;
};

}

BOOST_FUSION_ADAPT_STRUCT(
    ast::ast_struct,
    (int, number)
    (std::vector<int>, numbers)
)

namespace x3 = boost::spirit::x3;
using namespace std;

void parse( const std::string &data )
{
  string::const_iterator begin = data.begin();
  string::const_iterator end = data.end();

  unsigned n(0);

  auto f = [&n]( auto &ctx )
    {
      n = x3::_attr(ctx);
    };

  ast::ast_struct ast;
  bool r = x3::parse( begin, end, 
                      x3::int_[f] >> +( x3::omit[+x3::blank] >> x3::int_ ), ast );

  if ( r && begin == end )
  {
    cout << "n: " << n << ", ";
    std::copy(ast.numbers.begin(), ast.numbers.end(), 
              std::ostream_iterator<int>(std::cout << ast.numbers.size() << " elements: ", " "));
    cout << endl;
  }
  else
    cout << "Parse failed" << endl;
}

int main()
{
  parse( "3 1 2 3" );
  parse( "4 1 2 3 4" );
  return 0;
}

Running the code above (compiled with flags -std=c++14) outputs the expected result:

n: 3, 3 elements: 1 2 3 
n: 4, 4 elements: 1 2 3 4 

Now I am trying to have my Spirit X3 parser organized more or less the same way as the calc 9 example from Boost Spirit X3, but it does not work:

  • ast.hxx: defines the abstract syntax tree.
  • grammar.hxx: user interface exposing the parser methods.
  • grammar.cxx: instantiates the rules.
  • grammar_def.hxx: parser grammar definition.
  • config.hxx: parser configuration.
  • main.cxx: parser usage example.

ast.hxx:

#ifndef AST_HXX
#define AST_HXX

#include <vector>
#include <boost/fusion/include/adapt_struct.hpp>

namespace ast 
{

struct ast_struct
{
  int number;
  std::vector<int> numbers;
};

}

BOOST_FUSION_ADAPT_STRUCT(
    ast::ast_struct,
    (int, number)
    (std::vector<int>, numbers)
)

#endif

grammar.hxx:

#ifndef GRAMMAR_HXX
#define GRAMMAR_HXX

#include "ast.hxx"
#include <boost/spirit/home/x3.hpp>

namespace parser 
{

namespace x3 = boost::spirit::x3;

using my_rule_type = x3::rule<class my_rule_class, ast::ast_struct>;

BOOST_SPIRIT_DECLARE( my_rule_type );

const my_rule_type &get_my_rule();

}

#endif

grammar.cxx:

#include "grammar_def.hxx"
#include "config.hxx"

namespace parser 
{

BOOST_SPIRIT_INSTANTIATE( my_rule_type, iterator_type, context_type )

}

grammar_def.hxx:

#ifndef GRAMMAR_DEF_HXX
#define GRAMMAR_DEF_HXX

#include <iostream>
#include <boost/spirit/home/x3.hpp>
#include "grammar.hxx"
#include "ast.hxx"

namespace parser 
{
namespace x3 = boost::spirit::x3;

const my_rule_type  my_rule( "my_rule" );

unsigned n;

auto f = []( auto &ctx )
{
  n = x3::_attr(ctx);
};

auto my_rule_def =  x3::int_[f] >> +( x3::omit[+x3::blank] >> x3::int_ );

BOOST_SPIRIT_DEFINE( my_rule )

const my_rule_type &get_my_rule()
{
  return my_rule;
}

}

#endif

config.hxx:

#ifndef CONFIG_HXX
#define CONFIG_HXX

#include <string>
#include <boost/spirit/home/x3.hpp>

namespace parser 
{

namespace x3 = boost::spirit::x3;

using iterator_type = std::string::const_iterator;
using context_type = x3::unused_type;

}

#endif

main.cxx:

#include "ast.hxx"
#include "grammar.hxx"
#include "config.hxx"
#include <iostream>
#include <boost/spirit/home/x3.hpp>
#include <string>

namespace x3 = boost::spirit::x3;
using namespace std;

void parse( const std::string &data )
{
  parser::iterator_type begin = data.begin();
  parser::iterator_type end = data.end();

  ast::ast_struct ast;
  cout << "Parsing [" << string(begin,end) << "]" << endl;

  bool r = x3::parse( begin, end, parser::get_my_rule(), ast );

  if ( r && begin == end )
  {
    std::copy(ast.numbers.begin(), ast.numbers.end(), 
              std::ostream_iterator<int>(std::cout << ast.numbers.size() << " elements: ", " "));
    cout << endl;
  }
  else
    cout << "Parse failed" << endl;
}

int main()
{
  parse( "3 1 2 3" );
  parse( "4 1 2 3 4" );
  return 0;
}

Compiling main.cxx and grammar.cxx (flags: -std=c++14) and running the code above prints:

Parsing [3 1 2 3]
0 elements: 
Parsing [4 1 2 3 4]
0 elements: 

I apologize for the long source code, I tried to make it as small as possible.

Please notice I have some usage for the unsigned n global variable, it will be used with a custom repeat directive (see question here and one of the solutions here). In order to keep the question focused I removed the repeat part from this question, so even though I could remove the semantic action in this example, it is not a possible solution.

I would appreciate some help to get this issue uncovered, it is not clear to me why the code above does not work. Thank you in advance.

like image 392
Guilherme Schlinker Avatar asked Nov 26 '15 02:11

Guilherme Schlinker


1 Answers

I must admit actually reconstructing your sample was a bit too much work for me (call me lazy...).

However, I know the answer and a trick to make your life simpler.

The Answer

Semantic actions on a rule definition inhibit automatic attribute propagation. From the Qi docs (the same goes for X3, but I always lose the link to the docs):

r = p; Rule definition
This is equivalent to r %= p (see below) if there are no semantic actions attached anywhere in p.

r %= p; Auto-rule definition
The attribute of p should be compatible with the synthesized attribute of r. When p is successful, its attribute is automatically propagated to r's synthesized attribute.

The Trick

You can inject state (your n reference, in this case) using the x3::with<> directive. That way you don't have the namespace global (n) and can make the parser reentrant, threadsafe etc.

Here's my "simplist" take on things, in a single file:

namespace parsing {
    x3::rule<struct parser, ast::ast_struct> parser {"parser"};

    struct state_tag { };

    auto record_number = [](auto &ctx) { 
        unsigned& n = x3::get<state_tag>(ctx);
        n = x3::_attr(ctx); 
    };

    auto parser_def = x3::rule<struct parser_def, ast::ast_struct> {} 
                   %= x3::int_[record_number] >> +(x3::omit[+x3::blank] >> x3::int_);

    BOOST_SPIRIT_DEFINE(parser)
}

Tip: run the demo with = instead of the %= to see the difference in behaviour!

Note that get<state_tag>(ctx) returns a reference_wrapper<unsigned> just because we use the parser as follows:

void parse(const std::string &data) {
    using namespace std;

    ast::ast_struct ast;
    unsigned n;
    auto parser = x3::with<parsing::state_tag>(ref(n)) [parsing::parser] >> x3::eoi;

    if (x3::parse(data.begin(), data.end(), parser, ast)) {
        cout << "n: " << n << ", ";
        copy(ast.numbers.begin(), ast.numbers.end(), ostream_iterator<int>(cout << ast.numbers.size() << " elements: ", " "));
        cout << "\n";
    } else
        cout << "Parse failed\n";
}

Live Demo

Live On Coliru

#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/spirit/home/x3.hpp>
#include <iostream>

namespace ast {
    struct ast_struct {
        int number;
        std::vector<int> numbers;
    };
}

BOOST_FUSION_ADAPT_STRUCT(ast::ast_struct, number, numbers)

namespace x3 = boost::spirit::x3;

namespace parsing {
    x3::rule<struct parser, ast::ast_struct> parser {"parser"};

    struct state_tag { };

    auto record_number = [](auto &ctx) { 
        unsigned& n = x3::get<state_tag>(ctx); // note: returns reference_wrapper<T>
        n = x3::_attr(ctx); 
    };

    auto parser_def = x3::rule<struct parser_def, ast::ast_struct> {} 
                   %= x3::int_[record_number] >> +(x3::omit[+x3::blank] >> x3::int_);

    BOOST_SPIRIT_DEFINE(parser)
}

void parse(const std::string &data) {
    using namespace std;

    ast::ast_struct ast;
    unsigned n = 0;
    auto parser = x3::with<parsing::state_tag>(ref(n)) [parsing::parser] >> x3::eoi;

    if (x3::parse(data.begin(), data.end(), parser, ast)) {
        cout << "n: " << n << ", ";
        copy(ast.numbers.begin(), ast.numbers.end(), ostream_iterator<int>(cout << ast.numbers.size() << " elements: ", " "));
        cout << "\n";
    } else
        cout << "Parse failed\n";
}

int main() {
    parse("3 1 2 3");
    parse("4 1 2 3 4");
}

Prints

n: 3, 3 elements: 1 2 3 
n: 4, 4 elements: 1 2 3 4 
like image 58
sehe Avatar answered Oct 20 '22 00:10

sehe