Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to set custom stop words for sklearn CountVectorizer?

I'm trying to run LDA (Latent Dirichlet Allocation) on a non-English text dataset.

From sklearn's tutorial, there's this part where you count term frequency of the words to feed into the LDA:

tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                            max_features=n_features,
                            stop_words='english')

Which has built-in stop words feature which is only available for English I think. How could I use my own stop words list for this?

like image 683
troll Avatar asked Oct 19 '16 07:10

troll


1 Answers

You may just assign a frozenset of your own words to the stop_words argument, e.g.:

stop_words = frozenset(["word1", "word2","word3"])
like image 147
Wiktor Stribiżew Avatar answered Oct 25 '22 20:10

Wiktor Stribiżew