Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

split data frame based on integer index

Tags:

pandas

In pandas how do I split Series/dataframe into two Series/DataFrames where odd rows in one Series, even rows in different? Right now I am using

rng = range(0, n, 2)
odd_rows = df.iloc[rng]

This is pretty slow.

like image 226
user2426361 Avatar asked Jul 03 '13 20:07

user2426361


People also ask

How do you split a DataFrame into two parts based on condition?

In the above example, the data frame 'df' is split into 2 parts 'df1' and 'df2' on the basis of values of column 'Weight'. Method 2: Using Dataframe. groupby(). This method is used to split the data into groups based on some criteria.


2 Answers

Use slice:

In [11]: s = pd.Series([1,2,3,4])

In [12]: s.iloc[::2]  # even
Out[12]:
0    1
2    3
dtype: int64

In [13]: s.iloc[1::2]  # odd
Out[13]:
1    2
3    4
dtype: int64
like image 181
Andy Hayden Avatar answered Sep 24 '22 09:09

Andy Hayden


Here's some comparisions

In [100]: df = DataFrame(randn(100000,10))

simple method (but I think range makes this slow), but will work regardless of the index (e.g. does not have to be a numeric index)

In [96]: %timeit df.iloc[range(0,len(df),2)]
10 loops, best of 3: 21.2 ms per loop

The following require an Int64Index that is range based (which is easy to get, just reset_index()).

In [107]: %timeit df.iloc[(df.index % 2).astype(bool)]
100 loops, best of 3: 5.67 ms per loop

In [108]: %timeit df.loc[(df.index % 2).astype(bool)]
100 loops, best of 3: 5.48 ms per loop

make sure to give it index positions

In [98]: %timeit df.take(df.index % 2)
100 loops, best of 3: 3.06 ms per loop

same as above but no conversions on negative indicies

In [99]: %timeit df.take(df.index % 2,convert=False)
100 loops, best of 3: 2.44 ms per loop

This winner is @AndyHayden soln; this only works on a single dtype

In [118]: %timeit DataFrame(df.values[::2],index=df.index[::2])
10000 loops, best of 3: 63.5 us per loop
like image 45
Jeff Avatar answered Sep 24 '22 09:09

Jeff



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!