CountVectorizer(analyzer='char_wb') not working as expected

Question

I'm trying to use scikit-learn's CountVectorizer to count character 2-grams, ignoring spaces. In the docs it mentions the parameter analyzer which states

Whether the feature should be made of word or character n-grams. Option ‘char_wb’ creates character n-grams only from text inside word boundaries.

However, "char_wb" doesn't appear to work as I expected. For example:

corpus = [
    "The blue dog Blue",
    "Green the green cat",
    "The green mouse",
]

# CountVectorizer character 2-grams with word boundaries
vectorizer = CountVectorizer(analyzer='char_wb', ngram_range=(2, 2), min_df=1) 
X = vectorizer.fit_transform(corpus)
vectorizer.get_feature_names()
[' b',
 ' c',
 ' d',
 ' g',
 ' m',
 ' t',
 'at',
 'bl',
 'ca', ....

Notice the examples like ' b' which include a space. What gives?

Answer 1

I think this is a longstanding inaccuracy in the documentation, which you are welcome to help fix. It would be more correct to say that:

Option ‘char_wb’ creates character n-grams, but does not generate n-grams that cross word boundaries.

The change appears to have been made in this commit to ensure that; see the contributor's comment. It looks particularly awkward when comparing the bigrams output to that of analyzer='char', but when you increase to trigrams you will see that whitespace can begin or end an n-gram but cannot be in the middle. This helps to signify the word-initial or word-final nature of a feature without capturing noisy cross-word character n-grams. It also ensures that, unlike prior to that commit, all extracted n-grams have the same length!