Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Remove empty documents from DocumentTermMatrix in R topicmodels?

I am doing topic modelling using the topicmodels package in R. I am creating a Corpus object, doing some basic preprocessing, and then creating a DocumentTermMatrix:

corpus <- Corpus(VectorSource(vec), readerControl=list(language="en"))  corpus <- tm_map(corpus, tolower) corpus <- tm_map(corpus, removePunctuation) corpus <- tm_map(corpus, removeWords, stopwords("english")) corpus <- tm_map(corpus, stripWhitespace) corpus <- tm_map(corpus, removeNumbers) ...snip removing several custom lists of stopwords... corpus <- tm_map(corpus, stemDocument) dtm <- DocumentTermMatrix(corpus, control=list(minDocFreq=2, minWordLength=2)) 

And then performing LDA:

LDA(dtm, 30) 

This final call to LDA() returns the error

  "Each row of the input matrix needs to contain at least one non-zero entry".  

I assume this means that there is at least one document that has no terms in it after preprocessing. Is there an easy way to remove documents that contain no terms from a DocumentTermMatrix?

I looked in the documentation for the topicmodels package and found the function removeSparseTerms, which removes terms that do not appear in any document, but there is no analogue for removing documents.

like image 889
Bill M Avatar asked Dec 19 '12 01:12

Bill M


1 Answers

"Each row of the input matrix needs to contain at least one non-zero entry" 

The error means that sparse matrix contain a row without entries(words). one Idea is to compute the sum of words by row

rowTotals <- apply(dtm , 1, sum) #Find the sum of words in each Document dtm.new   <- dtm[rowTotals> 0, ]           #remove all docs without words 
like image 92
agstudy Avatar answered Sep 22 '22 15:09

agstudy