htmlcxx c++ crawling html

Question

I've read many different issues from different people regarding libraries to crawl HTML. I've decided to go with htmlcxx, because it looks simple and it's in the Ubuntu repository.. Anyhow, while playing with htmlcxx I was trying to achieve a simple task and grabbing the text between heading tags. Using the iterator, it->text() returns the tag itself and it->textClosing() returns the closing of the tag. My question is, how can I get at the data BETWEEN the tags? I mean, there must be a way, why make a library to crawl html and not have this functionality? If anyone can point me in the right direction, I'd appreciate it.

You can check out what I've done with svn so far with: svn co svn://yunices.dyndns.org/repository/nich/trunk .

or view through websvn: https://yunices.dyndns.org/

Here's the particular snippet in question:

void node::get_headings() {
    tree<htmlcxx::HTML::Node>::iterator it = dom.begin();
    tree<htmlcxx::HTML::Node>::iterator end = dom.end();
    for (; it != end; ++it) {
        static const boost::regex expression("[h|H][1-6]");
        if(boost::regex_search(it->tagName(), expression)) {
            it->parseAttributes();
            std::cout << it->text() << "<=>" << it->closingText() << std::endl;
            std::map<std::string, std::string> pairs = it->attributes();
            for ( std::map<std::string, std::string>::const_iterator iter = pairs.begin();
                    iter != pairs.end(); ++iter ) {
                std::cout << iter->first << ":" << iter->second << "\n";
            }
        }
    }
}

Answer 1

In most DOM libraries (and so in htmlcxx if I read the code correctly) the text of a tag is actually a node (or in the case of something like

<p> bla <p>blubb</p> blah </p>

more than one node).

You just have to iterate over all children of the tag and check that it's neither a comment nor a tag.