boost::split leaves empty tokens at the beginning and end of string - is this desired behaviour?

Question

Since I couldn't find anything on this in the documentation, I thought I ask it here. I have the following program (C++11):

#include <iostream> 
#include <boost/algorithm/string.hpp>

using namespace std;
using namespace boost;

int main () {
    string tmp = " #tag #tag1#tag2  #tag3 ####tag4   ";
    list<iterator_range<string::iterator> > matches;
    split( matches, tmp, is_any_of("\t #"), token_compress_on );

    for( auto match: matches ) {
            cout << "'" << match << "'\n";
    }
}

The output is:

''
'tag'
'tag1'
'tag2'
'tag3'
'tag4'
''

I would have thought that the token_compress_on option removes all empty tokens. The solution is, for example, to use boost::trim_if. Nevertheless I was wondering if this is the desired behaviour of boost::split, and why this is happening?

(g++ 4.6.3, boost 1.48)

Answer 1

boost::split always returns n + 1 tokens where n is number of separators in the input string. So don't be surprised when it returns 1 token when you pass it an empty string.

The rationale behind it is quite simple. Imagine that you are parsing a CSV file. You need to get the exactly same number of elements regardless whether the last token is empty or not.

It is much easier to remove the empty tokens than to guess whether they were supposed to be in the result or not. Credit

This behaviour is similar to python

>>> len("".split(','))
1

Answer 2

If eCompress argument is set to token_compress_on, adjacent separators are merged together. Otherwise, every two separators delimit a token.

Here

It does not remove the tokens only merges them.