Split dataframe into two on the basis of date

Question

I have dataset with 1000 rows like this

 Date,      Cost,         Quantity(in ton),    Source,          Unloading Station
    01/10/2015, 7,            5.416,               XYZ,           ABC

i want to split the data on the base of date. For e.g. till date 20.12.2016 is a training data and after that it is test data.

How should i split? Is it possible?

Answer 1

You can easily do that by converting your column to pandas to_datetime type and set it as index.

import pandas as pd
df['Date'] = pd.to_datetime(df['Date'])
df = df.set_index(df['Date'])
df = df.sort_index()

Once you have your data in this format, you can simply use date as index for creating partition as follows:

# create train test partition
train = df['2015-01-10':'2016-12-20']
test  = df['2016-12-21':]
print('Train Dataset:',train.shape)
print('Test Dataset:',test.shape)

Answer 2

assuming that your data set is pandas data frame and that Date column is of datetime dtype:

split_date = pd.datetime(2016,12,20)

df_training = df.loc[df['Date'] <= split_date]
df_test = df.loc[df['Date'] > split_date]

Answer 3

If your date is in standard python datetime format ie. '2016-06-23 23:00:00', you can use the code below

split_date ='2016-06-23 23:00:00' train_data = train_data.loc[train_data['Date'] <= split_date] validation_data = train_data.loc[train_data['Date'] > split_date]