Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to impute column values on Dask Dataframe?

Tags:

I would like to impute negative values of Dask Dataframe, with pandas i use this code:

df.loc[(df.column_name < 0),'column_name'] = 0
like image 406
ambigus9 Avatar asked Mar 25 '18 15:03

ambigus9


1 Answers

I think need dask.dataframe.Series.clip_lower:

ddf['B'] = ddf['B'].clip_lower(0)

Sample:

import pandas as pd

df = pd.DataFrame({'F':list('abcdef'),
                   'B':[-4,5,4,-5,5,4],
                   'A':list('aaabbb')})

print (df)
   A  B  F
0  a -4  a
1  a  5  b
2  a  4  c
3  b -5  d
4  b  5  e
5  b  4  f

from dask import dataframe as dd 
ddf = dd.from_pandas(df, npartitions=3)
#print (ddf)

ddf['B'] = ddf['B'].clip_lower(0)
print (ddf.compute())
   A  B  F
0  a  0  a
1  a  5  b
2  a  4  c
3  b  0  d
4  b  5  e
5  b  4  f

For more general solution use dask.dataframe.Series.mask`:

ddf['B'] = ddf['B'].mask(ddf['B'] > 0, 3)
print (ddf.compute())
   A  B  F
0  a -4  a
1  a  3  b
2  a  3  c
3  b -5  d
4  b  3  e
5  b  3  f
like image 144
jezrael Avatar answered Sep 20 '22 13:09

jezrael