Menu
I have been trying word2vec for a while now using the gensim's word2vec library. My question is do I have to remove stopwords from my input text? Because, based on my initial experimental results, I could see words like 'of', 'when'.. (stopwords) popping up when I do a model.most_similar('someword')..?
But I didn't see anywhere referring that stop word removal is necessary with word2vec? Does the word2vec is supposed to handle stop words even if you don't remove them?
What are the must do pre processing things (like for topic modeling, it's almost a must that you should do stopword removal)?
294
word2vec can learn words those occur in the same context. So, I recommend you to train a model by removing stop words and then train a model without stop words and check which one is performing good.
Stop words are available in abundance in any human language. By removing these words, we remove the low-level information from our text in order to give more focus to the important information.
To remove stop words from a sentence, you can divide your text into words and then remove the word if it exits in the list of stop words provided by NLTK. In the script above, we first import the stopwords collection from the nltk. corpus module. Next, we import the word_tokenize() method from the nltk.
In a lot of tutorials about Machine Learning applied to text, you may read that removing stop words is a necessary pre-processing step. Apparently, removing stop words is not only necessary, but is also a must do.
I think by removing stop words your results will become better. Its because of frequent words like 'the', 'of', 'is' are not very important until or unless you are dealing some sort of sentence structures ( or syntactic structures). word2vec can learn words those occur in the same context.
Removing stop words with NLTK. The following program removes stop words from a piece of text: from nltk.corpus import stopwords. from nltk.tokenize import word_tokenize. example_sent = "This is a sample sentence, showing off the stop words filtration.". stop_words = set(stopwords.words('english'))
As others mentioned before, it really depends on what you want to do, and the best answer cannot be found in personal opinions, but in experiments. Stop words may play a role in word embedding by associating related words through their relationship to some of those stop words.
However, after the removal of stop words, the review became positive, which is not the reality. Thus, the removal of stop words can be problematic here. Tasks like text classification do not generally need stop words as the other words present in the dataset are more important and give the general idea of the text.
Gensim's implementation is based on the original Tomas Mikolov model of word2vec, then it downsamples all frequent words automatically based on frequency.
As stated in the paper:
We show that subsampling of frequent words during training results in a significant speedup (around 2x - 10x), and improves accuracy of the representations of less frequent words.
What it means is that these words are sometimes not considered in the window of the words to be predicted. The sample parameter which defaults to 0.001 is used as a parameter to prune out those words. If you want to remove some specific stopwords which would not be removed based on its frequency, you can do that.
Summary : The result would not make any significant difference if you do stop words removal.
111
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
