apply function to textreuse corpus

Question

I have a data frame as follows:

df<-data.frame(revtext=c('the dog that chased the cat', 'the dog which chased the cat', 'World Cup Hair 2014 very funny.i can change', 'BowBow', 'this is'), rid=c('r01','r02','r03','r04','r05'), stringsAsFactors = FALSE)

                             revtext        rid
             the dog that chased the cat    r01
             the dog which chased the cat   r02
World Cup Hair 2014 very funny.i can change r03
             Bow Bow                        r04
             this is                        r05

I'm using the package textreuse to convert df to a corpus doing:

#install.packages(textreuse)
library(textreuse)
d<-df$revtext
names(d)<-df$rid
corpus <- TextReuseCorpus(text = d,
                      tokenizer = tokenize_character, k=3,
                      progress = FALSE,
                      keep_tokens = TRUE)

where tokenize_character is a function I programmed as:

 tokenize_character <- function(document, k) {
                       shingles<-c()
                 for( i in 1:( nchar(document) - k + 1 ) ) {
                         shingles[i] <- substr(document,start=i,stop= (i+k-1))
                     }
return( unique(shingles) )  
}   

However, I'm prompted with some warnings: Skipping document with ID 'r04' because it has too few words to create at least two n-grams with n = 3.. But note that my tokenizer works on a character level. The text of r04 is long enough. In fact, if we run tokenize_character('BowBow',3) we get: "Bow" "owB" "wBo" as desired.

Note also that for r01, TextReuseCorpus is working as it is supposed, returning: tokens(corpus)$`r01= "the" "he " "e d" " do" "dog" "og " "g t" " th" "tha" "hat" "at " "t c" " ch" "cha" "has" "ase" "sed" "ed " "d t" "e c" " ca" "cat"

Any suggestions? I don't know what I'm missing here.

Answer 1

From the details section of textreuse::TextReuseCorpus documentation:

If skip_short = TRUE, this function will skip very short or empty documents. A very short document is one where there are two few words to create at least two n-grams. For example, if five-grams are desired, then a document must be at least six words long. If no value of n is provided, then the function assumes a value of n = 3.

From this, we know documents having < 4 words will be skipped as short documents (as n=3 in your example), which is what we see for r04 & r05 that have 1 & 2 words, respectively. To not skip these documents, you can use skip_short = F which will return the output as intended:

corpus <- TextReuseCorpus(text = d, tokenizer = tokenize_character, k=3,
                      skip_short = F, progress = FALSE, keep_tokens = TRUE)
tokens(corpus)$r04
[1] "Bow" "owB" "wBo"