Numpy: Fix array with rows of different lengths by filling the empty elements with zeros

Question

The functionality I am looking for looks something like this:

data = np.array([[1, 2, 3, 4],                  [2, 3, 1],                  [5, 5, 5, 5],                  [1, 1]])  result = fix(data) print result  [[ 1.  2.  3.  4.]  [ 2.  3.  1.  0.]  [ 5.  5.  5.  5.]  [ 1.  1.  0.  0.]] 

These data arrays I'm working with are really large so I would really appreciate the most efficient solution.

Edit: Data is read in from disk as a python list of lists.

Answer 1

This could be one approach -

def numpy_fillna(data):     # Get lengths of each row of data     lens = np.array([len(i) for i in data])      # Mask of valid places in each row     mask = np.arange(lens.max()) < lens[:,None]      # Setup output array and put elements from data into masked positions     out = np.zeros(mask.shape, dtype=data.dtype)     out[mask] = np.concatenate(data)     return out 

Sample input, output -

In [222]: # Input object dtype array      ...: data = np.array([[1, 2, 3, 4],      ...:                  [2, 3, 1],      ...:                  [5, 5, 5, 5, 8 ,9 ,5],      ...:                  [1, 1]])  In [223]: numpy_fillna(data) Out[223]:  array([[1, 2, 3, 4, 0, 0, 0],        [2, 3, 1, 0, 0, 0, 0],        [5, 5, 5, 5, 8, 9, 5],        [1, 1, 0, 0, 0, 0, 0]], dtype=object) 

Answer 2

You could use pandas instead of numpy:

In [1]: import pandas as pd  In [2]: df = pd.DataFrame([[1, 2, 3, 4],    ...:                    [2, 3, 1],    ...:                    [5, 5, 5, 5],    ...:                    [1, 1]], dtype=float)   In [3]: df.fillna(0.0).values Out[3]:  array([[ 1.,  2.,  3.,  4.],        [ 2.,  3.,  1.,  0.],        [ 5.,  5.,  5.,  5.],        [ 1.,  1.,  0.,  0.]])