Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

stratified splitting the data

I have a large data set and like to fit different logistic regression for each City, one of the column in my data. The following 70/30 split works without considering City group.

indexes <- sample(1:nrow(data), size = 0.7*nrow(data))

train <- data[indexes,]
test <- data[-indexes,]

But this does not guarantee the 70/30 split for each city.

lets say that I have City A and City B, where City A has 100 rows, and City B has 900 rows, totaling 1000 rows. Splitting the data with above code will give me 700 rows for train and 300 for test data, but it does not guarantee that i will have 70 rows for City A, and 630 rows for City B in the train data. How do i do that?

Once i have the training data split-ed to 70/30 fashion for each city,i will run logistic regression for each city ( I know how to do this once i have the train data)

like image 929
user35577 Avatar asked Dec 25 '13 21:12

user35577


People also ask

Why is it important to use stratified sampling while splitting data?

Stratified random sampling is typically used by researchers when trying to evaluate data from different subgroups or strata. It allows them to quickly obtain a sample population that best represents the entire population being studied.

What is stratified train test split?

As such, it is desirable to split the dataset into train and test sets in a way that preserves the same proportions of examples in each class as observed in the original dataset. This is called a stratified train-test split.

What does it mean to stratify your data?

Stratification is defined as the act of sorting data, people, and objects into distinct groups or layers. It is a technique used in combination with other data analysis tools. When data from a variety of sources or categories have been lumped together, the meaning of the data can be difficult to see.

Which method is used to split the data?

The simplest and probably the most common strategy to split such a dataset is to randomly sample a fraction of the dataset. For example, 80% of the rows of the dataset can be randomly chosen for training and the remaining 20% can be used for testing.


1 Answers

Try createDataPartition from caret package. Its document states: By default, createDataPartition does a stratified random split of the data.

library(caret)
train.index <- createDataPartition(Data$Class, p = .7, list = FALSE)
train <- Data[ train.index,]
test  <- Data[-train.index,]

it can also be used for stratified K-fold like:

ctrl <- trainControl(method = "repeatedcv",
                     repeats = 3,
                     ...)
# when calling train, pass this train control
train(...,
      trControl = ctrl,
      ...)

check out caret document for more details

like image 103
muon Avatar answered Oct 22 '22 04:10

muon