Find duplicate values in R [duplicate]

Question

You could use table, i.e.

n_occur <- data.frame(table(vocabulary$id))

gives you a data frame with a list of ids and the number of times they occurred.

n_occur[n_occur$Freq > 1,]

tells you which ids occurred more than once.

vocabulary[vocabulary$id %in% n_occur$Var1[n_occur$Freq > 1],]

returns the records with more than one occurrence.

This will give you duplicate rows:

vocabulary[duplicated(vocabulary$id),]

This will give you the number of duplicates:

dim(vocabulary[duplicated(vocabulary$id),])[1]

Example:

vocabulary2 <-rbind(vocabulary,vocabulary[1,]) #creates a duplicate at the end
vocabulary2[duplicated(vocabulary2$id),]
#            id year    sex education vocabulary
#21639 20040001 2004 Female         9          3
dim(vocabulary2[duplicated(vocabulary2$id),])[1]
#[1] 1 #=1 duplicate

EDIT

OK, with the additional information, here's what you should do: duplicated has a fromLast option which allows you to get duplicates from the end. If you combine this with the normal duplicated, you get all duplicates. The following example adds duplicates to the original vocabulary object (line 1 is duplicated twice and line 5 is duplicated once). I then use table to get the total number of duplicates per ID.

#Create vocabulary object with duplicates
voc.dups <-rbind(vocabulary,vocabulary[1,],vocabulary[1,],vocabulary[5,])

#List duplicates
dups <-voc.dups[duplicated(voc.dups$id)|duplicated(voc.dups$id, fromLast=TRUE),]
dups
#            id year    sex education vocabulary
#1     20040001 2004 Female         9          3
#5     20040008 2004   Male        14          1
#21639 20040001 2004 Female         9          3
#21640 20040001 2004 Female         9          3
#51000 20040008 2004   Male        14          1

#Count duplicates by id
table(dups$id)
#20040001 20040008 
#       3        2 

Here, I summarize a few ways which may return different results to your question, so be careful:

# First assign your "id"s to an R object.
# Here's a hypothetical example:
id <- c("a","b","b","c","c","c","d","d","d","d")

#To return ALL MINUS ONE duplicated values:
id[duplicated(id)]
## [1] "b" "c" "c" "d" "d" "d"

#To return ALL duplicated values by specifying fromLast argument:
id[duplicated(id) | duplicated(id, fromLast=TRUE)]
## [1] "b" "b" "c" "c" "c" "d" "d" "d" "d"

#Yet another way to return ALL duplicated values, using %in% operator:
id[ id %in% id[duplicated(id)] ]
## [1] "b" "b" "c" "c" "c" "d" "d" "d" "d"

Hope these help. Good luck.

Here's a data.table solution that will list the duplicates along with the number of duplications (will be 1 if there are 2 copies, and so on - you can adjust that to suit your needs):

library(data.table)
dt = data.table(vocabulary)

dt[duplicated(id), cbind(.SD[1], number = .N), by = id]

A terser way, either with rev :

x[!(!duplicated(x) & rev(!duplicated(rev(x))))]

... rather than fromLast:

x[!(!duplicated(x) & !duplicated(x, fromLast = TRUE))]

... and as a helper function to provide either logical vector or elements from original vector :

duplicates <- function(x, as.bool = FALSE) {
    is.dup <- !(!duplicated(x) & rev(!duplicated(rev(x))))
    if (as.bool) { is.dup } else { x[is.dup] }
}

Treating vectors as data frames to pass to table is handy but can get difficult to read, and the data.table solution is fine but I'd prefer base R solutions for dealing with simple vectors like IDs.