Remove single occurrences of words in vocabulary TF-IDF

Question

I am attempting to remove words that occur once in my vocabulary to reduce my vocabulary size. I am using the sklearn TfidfVectorizer() and then the fit_transform function on my data frame.

tfidf = TfidfVectorizer()  
tfs = tfidf.fit_transform(df['original_post'].values.astype('U')) 

My first thought is the preprocessor field in the tfidf vectorizer or using the preprocessing package before machine learning.

Any tips or links to further implementation?

Answer 1

you are looking for min_df param (minimum frequency), from the documentation of scikit-learn TfidfVectorizer:

min_df : float in range [0.0, 1.0] or int, default=1

When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. This value is also called cut-off in the literature. If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None.

# remove words occuring less than 5 times
tfidf = TfidfVectorizer(min_df=5)

you can also remove common words:

# remove words occuring in more than half the documents
tfidf = TfidfVectorizer(max_df=0.5)

you can also remove stopwords like this:

tfidf = TfidfVectorizer(stop_words='english')