Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to replace None only with empty string using pandas?

Tags:

python

pandas

the code below generates a df:

import pandas as pd
from datetime import datetime as dt
import numpy as np

dates = [dt(2014, 1, 2, 2), dt(2014, 1, 2, 3), dt(2014, 1, 2, 4), None]
strings1 = ['A', 'B',None, 'C']
strings2 = [None, 'B','C', 'C']
strings3 = ['A', 'B','C', None]
vals = [1.,2.,np.nan, 4.]
df = pd.DataFrame(dict(zip(['A','B','C','D','E'],
                           [strings1, dates, strings2, strings3, vals])))



+---+------+---------------------+------+------+-----+
|   |  A   |          B          |  C   |  D   |  E  |
+---+------+---------------------+------+------+-----+
| 0 | A    | 2014-01-02 02:00:00 | None | A    | 1   |
| 1 | B    | 2014-01-02 03:00:00 | B    | B    | 2   |
| 2 | None | 2014-01-02 04:00:00 | C    | C    | NaN |
| 3 | C    | NaT                 | C    | None | 4   |
+---+------+---------------------+------+------+-----+

I would like to replace all None (real None in python, not str) inside with ''(empty string).

The expected df is

+---+---+---------------------+---+---+-----+
|   | A |          B          | C | D |  E  |
+---+---+---------------------+---+---+-----+
| 0 | A | 2014-01-02 02:00:00 |   | A | 1   |
| 1 | B | 2014-01-02 03:00:00 | B | B | 2   |
| 2 |   | 2014-01-02 04:00:00 | C | C | NaN |
| 3 | C | NaT                 | C |   | 4   |
+---+---+---------------------+---+---+-----+

what I did is

df = df.replace([None], [''], regex=True)

But I got

+---+---+---------------------+---+------+---+
|   | A |          B          | C |  D   | E |
+---+---+---------------------+---+------+---+
| 0 | A | 1388628000000000000 |   | A    | 1 |
| 1 | B | 1388631600000000000 | B | B    | 2 |
| 2 |   | 1388635200000000000 | C | C    |   |
| 3 | C |                     | C |      | 4 |
+---+---+---------------------+---+------+---+

  1. all the dates becomes big numbers
  2. Even NaT and NaN are replaced, which I don't want.

How can I achieve that correctly and efficently?

like image 323
Jackson Tale Avatar asked Jul 08 '15 14:07

Jackson Tale


1 Answers

This is sufficient

df.fillna("",inplace=True)
df
Out[142]: 
   A                    B  C  D  E
0  A  2014-01-02 02:00:00     A  1
1  B  2014-01-02 03:00:00  B  B  2
2     2014-01-02 04:00:00  C  C   
3  C                       C     4

edit 2021-07-26 complete response following @dWitty's comment

If you really want to keep Nat and NaN values on other than text, you just need fill Na for your text column In your exemple this is A, C, D

You just send a dict of replacement value for your columns. value can be differents for each column. For your case you just need construct the dict

# default values to replace NA (None)
# values = {"A": "", "C": "", "D": ""}
values = (dict([[e,""] for e in ['A','C','D']]))
df.fillna(value=values, inplace=True)
df
Out[142]: 
   A                   B  C  D    E
0  A 2014-01-02 02:00:00     A  1.0
1  B 2014-01-02 03:00:00  B  B  2.0
2    2014-01-02 04:00:00  C  C  NaN
3  C                 NaT  C     4.0
like image 182
GeoStoneMarten Avatar answered Sep 29 '22 13:09

GeoStoneMarten