Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does createDataPartition function from caret package split data?

From the documentation:

For bootstrap samples, simple random sampling is used.

For other data splitting, the random sampling is done within the levels of y when y is a factor in an attempt to balance the class distributions within the splits.

For numeric y, the sample is split into groups sections based on percentiles and sampling is done within these subgroups.

For createDataPartition, the number of percentiles is set via the groups argument.

I don't understand why this "balance" thing is needed. I think I understand it superficially, but any additional insight would be really helpful.

like image 364
happy_sisyphus Avatar asked Nov 20 '16 21:11

happy_sisyphus


1 Answers

It means, if you have a data set ds with 10000 rows

set.seed(42)
ds <- data.frame(values = runif(10000))

with 2 "classes" with unequal distribution (9000 vs 1000)

ds$class <- c(rep(1, 9000), rep(2, 1000))
ds$class <- as.factor(ds$class)
table(ds$class)
#    1    2 
# 9000 1000 

you can create a sample, which tries to maintain the ratio / "balance" of the factor classes.

dpart <- createDataPartition(ds$class, p = 0.1, list = F)
dsDP <- ds[dpart, ]
table(dsDP$class)
#   1   2 
# 900 100 
like image 97
loki Avatar answered Sep 18 '22 12:09

loki