Getting boost::spirit::qi to use stl containers

Question

I'm trying to parse something with boost.spirit's qi library, and I'm running into an issue. According to the spirit docs, a >> b should produce something with the type tuple<A, B>. But this is a boost::tuple (aka fusion vector), and not a std::tuple (which I want).

Is there any easy way to make this conversion between boost::tuple => std::tuple?

The same documentation page says that *a should produce something with the type vector<A>. This seems to be producing a std::vector<A> (or some kind of boost::vector<A> that can implicitly convert to a std::vector<A>). I just wanted to know if this same behavior was available for tuples.

Answer 1

SHORT ANSWER:

Use #include <boost/fusion/adapted/std_tuple.hpp>.

MORE COMPLETE ANSWER:

As you can see here:

In the attribute tables, we will use vector<A> and tuple<A, B...> as placeholders only. The notation of vector<A> stands for any STL container holding elements of type A and the notation tuple<A, B...> stands for any Boost.Fusion sequence holding A, B, ... etc. elements. Finally, Unused stands for unused_type.

So when a parser/generator has an attribute of tuple<A,B...> you can use any fusion sequence (such as fusion::vector or fusion::list) or anything that can be adapted to a fusion sequence (such as boost::array, boost::tuple, std::pair, std::tuple, your own struct using BOOST_FUSION_ADAPT_STRUCT).

And when it has vector<A> you can use std::vector, std::list, and even std::map if your elements are pairs. You can also use your own struct if you also specialize several customization points (at least is_container, container_value and push_back_container in boost::spirit::traits).

std::pair
In order to be able to use std::pair with spirit you just need to add a single header:

#include <boost/fusion/include/std_pair.hpp>
...
qi::rule<Iterator,std::pair<int,double>()> rule = 
    qi::int_ >> qi::lit(',') >> qi::double_;

std::tuple
Starting with boost 1.48.0 you can do the same for std::tuple:

#include <boost/fusion/adapted/std_tuple.hpp> 
...
qi::rule<Iterator,std::tuple<int,std::string,double>()> rule =
    qi::int_ >> qi::lit(',') >> +~qi::char_(',') >> qi::lit(',') >> qi::double_;

Your own struct
You can adapt your custom struct very easily with the help of BOOST_FUSION_ADAPT_STRUCT:

#include <boost/fusion/include/adapt_struct.hpp>
...
struct normal_struct
{
    int integer;
    double real;
};

BOOST_FUSION_ADAPT_STRUCT(
    normal_struct,
    (int, integer)
    (double, real)
)
...
qi::rule<Iterator,normal_struct()> rule =
    qi::int_ >> qi::lit(',') >> qi::double_;

There is one known limitation though, when you try to use a struct that has a single element that is also a container compilation fails unless you add qi::eps >> ... to your rule.

struct struct_with_single_element_container
{
    std::vector<int> cont;
};

BOOST_FUSION_ADAPT_STRUCT(
    struct_with_single_element_container,
    (std::vector<int>, cont)
)
...
qi::rule<Iterator,struct_with_single_element_container()> rule =
    qi::eps >> qi::int_%qi::lit(',');

std::map
You can simply use std::map as a container of std::pairs. Keep in mind though that if there are repeated keys in your input, only the first one will be inserted to the map (if you use multimap everything will be inserted of course):

#include <boost/fusion/include/std_pair.hpp>
...
qi::rule<std::string::const_iterator, std::pair<double,int>()> pair_rule = 
    qi::double_ >> qi::lit('=') >> qi::int_;
qi::rule<std::string::const_iterator, std::map<double,int>()> rule = 
    pair_rule%qi::lit(',');
//You can also use
//qi::rule<std::string::const_iterator, std::map<double,int>()> rule =
    //(qi::double_ >> qi::lit('=') >> qi::int_)%qi::lit(',');

Your own struct as a container
Using spirit's customization points you can also make your struct behave as if it were a container when dealing with attributes. The minimum you need to specialize are is_container, container_value and push_back_container. Here are a couple of examples:

---

The first one is rather simple (and silly). It makes your struct have an attribute compatible with std::vector<int>. Every time an int is parsed it is added to the total in the accumulator. You can find less silly approaches here and here (in the "old answer").

struct accumulator
{
    accumulator(): total(){}
    int total;
};

namespace boost{ namespace spirit{ namespace traits
{
    template<>
    struct is_container<accumulator> : boost::mpl::true_
    {};

    template<>
    struct container_value<accumulator>
    {
        typedef int type;
    };

    template<>
    struct push_back_container<accumulator,int>
    {
        static bool call(accumulator& c, int val)
        {
            c.total+=val;
            return true;
        }
    };
}}}
...
qi::rule<Iterator,accumulator()> rule =
    qi::int_%qi::lit(',');

---

The second one is a little more complex (not much). It makes your struct have an attribute compatible with std::vector<boost::variant<int,std::string> >. When an int is parsed it is added to the ints container in the distributor, similarly strings are stored in the strings container. Examples using this (1, 2 and 3).

struct distributor
{
    distributor():ints(),strings(){}
    std::vector<int> ints;
    std::vector<std::string> strings;
};

namespace boost{ namespace spirit{ namespace traits
{
    template<>
    struct is_container<distributor> : boost::mpl::true_
    {};

    template<>
    struct container_value<distributor>
    {
        typedef boost::variant<int,std::string> type;
    };

    template<>
    struct push_back_container<distributor,int>
    {
        static bool call(distributor& c, int val)
        {
            c.ints.push_back(val);
            return true;
        }
    };

    template<>
    struct push_back_container<distributor,std::string>
    {
        static bool call(distributor& c, std::string const& val)
        {
            c.strings.push_back(val);
            return true;
        }
    };
}}}
...
qi::rule<std::string::const_iterator, std::string()> string_rule = 
    +~qi::char_(',');
qi::rule<std::string::const_iterator, distributor()> rule = 
    (qi::int_ | string_rule)%qi::lit(',');

---

All the tests in a single cpp file

#include <iostream>
#include <string>
#include <utility>
#include <tuple>
#include <list>
#include <vector>
#include <map>

#include <boost/fusion/include/std_pair.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/adapted/std_tuple.hpp> 

#include <boost/spirit/include/qi.hpp>

#include <boost/variant.hpp>

namespace qi=boost::spirit::qi;

struct normal_struct
{
    int integer;
    double real;
};

struct struct_with_single_element_container
{
    std::vector<int> cont;
};

BOOST_FUSION_ADAPT_STRUCT(
    normal_struct,
    (int, integer)
    (double, real)
)

BOOST_FUSION_ADAPT_STRUCT(
    struct_with_single_element_container,
    (std::vector<int>, cont)
)

struct accumulator
{
    accumulator(): total(){}
    int total;
};

namespace boost{ namespace spirit{ namespace traits
{
    template<>
    struct is_container<accumulator> : boost::mpl::true_
    {};

    template<>
    struct container_value<accumulator>
    {
        typedef int type;
    };

    template<>
    struct push_back_container<accumulator,int>
    {
        static bool call(accumulator& c, int val)
        {
            c.total+=val;
            return true;
        }
    };
}}}

struct distributor
{
    distributor():ints(),strings(){}
    std::vector<int> ints;
    std::vector<std::string> strings;
};

namespace boost{ namespace spirit{ namespace traits
{
    template<>
    struct is_container<distributor> : boost::mpl::true_
    {};

    template<>
    struct container_value<distributor>
    {
        typedef boost::variant<int,std::string> type;
    };

    template<>
    struct push_back_container<distributor,int>
    {
        static bool call(distributor& c, int val)
        {
            c.ints.push_back(val);
            return true;
        }
    };

    template<>
    struct push_back_container<distributor,std::string>
    {
        static bool call(distributor& c, std::string const& val)
        {
            c.strings.push_back(val);
            return true;
        }
    };
}}}


int main()
{
    {
        std::pair<int,double> parsed;
        qi::rule<std::string::const_iterator, std::pair<int,double>()> rule = 
                    qi::int_ >> qi::lit(',') >> qi::double_;
        std::string test="1,2.5";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "First: " << parsed.first << ", Second: " << parsed.second << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

    {
        std::tuple<int,std::string,double> parsed;
        qi::rule<std::string::const_iterator, std::tuple<int,std::string,double>()> rule = 
                    qi::int_ >> qi::lit(',') >> +~qi::char_(',') >> qi::lit(',') >> qi::double_;
        std::string test="1,abc,2.5";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "get<0>: " << std::get<0>(parsed) << ", get<1>: " << std::get<1>(parsed) << ", get<2>: " << std::get<2>(parsed) << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

    {
        normal_struct parsed;
        qi::rule<std::string::const_iterator, normal_struct()> rule = 
                    qi::int_ >> qi::lit(',') >> qi::double_;
        std::string test="1,2.5";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "integer: " << parsed.integer << ", real: " << parsed.real << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

    {
        struct_with_single_element_container parsed;
        //there is a problem when you have a struct with a single element container, the workaround is simply adding qi::eps to the rule
        qi::rule<std::string::const_iterator, struct_with_single_element_container()> rule = 
                    qi::eps >> qi::int_%qi::lit(','); 
        std::string test="1,2";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "[0]: " << parsed.cont[0] << ", [1]: " << parsed.cont[1] << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

    {
        std::list<int> parsed;
        qi::rule<std::string::const_iterator, std::list<int>()> rule = 
                    qi::int_%qi::lit(',');
        std::string test="1,2";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "front: " << parsed.front() << ", back: " << parsed.back() << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

    {
        std::map<double,int> parsed;
        qi::rule<std::string::const_iterator, std::pair<double,int>()> pair_rule = 
                    qi::double_ >> qi::lit('=') >> qi::int_;
        qi::rule<std::string::const_iterator, std::map<double,int>()> rule = 
                    pair_rule%qi::lit(',');
        std::string test="2.5=1,3.5=2";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "map[2.5]: " << parsed[2.5] << ", map[3.5]: " << parsed[3.5] << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

    {
        accumulator parsed;
        qi::rule<std::string::const_iterator, accumulator()> rule = 
                    qi::int_%qi::lit(',');
        std::string test="1,2,3";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "total: " << parsed.total << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

    {
        distributor parsed;
        qi::rule<std::string::const_iterator, std::string()> string_rule = 
                    +~qi::char_(',');
        qi::rule<std::string::const_iterator, distributor()> rule = 
                    (qi::int_ | string_rule)%qi::lit(',');
        std::string test="abc,1,2,def,ghi,3,jkl";
        std::string::const_iterator iter=test.begin(), end=test.end();
        bool result = qi::parse(iter,end,rule,parsed);
        if(result && iter==end)
        {
            std::cout << "Success." << std::endl;
            std::cout << "ints" << std::endl;
            for(auto val: parsed.ints)
                std::cout << val << std::endl;
            std::cout << "strings" << std::endl;
            for(const auto& val: parsed.strings)
                std::cout << val << std::endl;
        }
        else
        {
            std::cout << "Failure." << std::endl;
            std::cout << "Unparsed: " << std::string(iter,end) << std::endl;
        }
    }

}