Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Numpy: Drop rows with all nan or 0 values

I'd like to drop all values from a table if the rows = nan or 0.

I know there's a way to do this using pandas i.e pandas.dropna(how = 'all') but I'd like a numpy method to remove rows with all nan or 0.

Is there an efficient implementation of this?

like image 637
Black Avatar asked Feb 26 '14 05:02

Black


3 Answers

import numpy as np

a = np.array([
    [1, 0, 0],
    [0, np.nan, 0],
    [0, 0, 0],
    [np.nan, np.nan, np.nan],
    [2, 3, 4]
])

mask = np.all(np.isnan(a) | np.equal(a, 0), axis=1)
a[~mask]
like image 196
HYRY Avatar answered Oct 23 '22 07:10

HYRY


This will remove all rows which are all zeros, or all nans:

mask = np.all(np.isnan(arr), axis=1) | np.all(arr == 0, axis=1)
arr = arr[~mask]

And this will remove all rows which are all either zeros or nans:

mask = np.all(np.isnan(arr) | arr == 0, axis=1)
arr = arr[~mask]
like image 7
Jaime Avatar answered Oct 23 '22 05:10

Jaime


In addition: if you want to drop rows if a row has a nan or 0 in any single value

a = np.array([
    [1, 0, 0],
    [1, 2, np.nan],
    [np.nan, np.nan, np.nan],
    [2, 3, 4]
])

mask = np.any(np.isnan(a) | np.equal(a, 0), axis=1)
a[~mask]

Output

array([[ 2.,  3.,  4.]])
like image 4
Greg Avatar answered Oct 23 '22 07:10

Greg