Why is numpy.array() is sometimes very slow?

Question

I'm using the numpy.array() function to create numpy.float64 ndarrays from lists.

I noticed that this is very slow when either the list contains None or a list of lists is provided.

Below are some examples with times. There are obvious workarounds but why is this so slow?

Examples for list of None:

### Very slow to call array() with list of None
In [3]: %timeit numpy.array([None]*100000, dtype=numpy.float64)
1 loops, best of 3: 240 ms per loop

### Problem doesn't exist with array of zeroes
In [4]: %timeit numpy.array([0.0]*100000, dtype=numpy.float64)
100 loops, best of 3: 9.94 ms per loop

### Also fast if we use dtype=object and convert to float64
In [5]: %timeit numpy.array([None]*100000, dtype=numpy.object).astype(numpy.float64)
100 loops, best of 3: 4.92 ms per loop

### Also fast if we use fromiter() insead of array()
In [6]: %timeit numpy.fromiter([None]*100000, dtype=numpy.float64)
100 loops, best of 3: 3.29 ms per loop

Examples for list of lists:

### Very slow to create column matrix
In [7]: %timeit numpy.array([[0.0]]*100000, dtype=numpy.float64)
1 loops, best of 3: 353 ms per loop

### No problem to create column vector and reshape
In [8]: %timeit numpy.array([0.0]*100000, dtype=numpy.float64).reshape((-1,1))
100 loops, best of 3: 10 ms per loop

### Can use itertools to flatten input lists
In [9]: %timeit numpy.fromiter(itertools.chain.from_iterable([[0.0]]*100000),dtype=numpy.float64).reshape((-1,1))
100 loops, best of 3: 9.65 ms per loop

Answer 1

I've reported this as a numpy issue. The report and patch files are here:

https://github.com/numpy/numpy/issues/3392

After patching:

# was 240 ms, best alternate version was 3.29
In [5]: %timeit numpy.array([None]*100000)
100 loops, best of 3: 7.49 ms per loop

# was 353 ms, best alternate version was 9.65
In [6]: %timeit numpy.array([[0.0]]*100000)
10 loops, best of 3: 23.7 ms per loop

Answer 2

My guess would be that the code for converting lists just calls float on everything. If the argument defines __float__, we call that, otherwise we treat it like a string (throwing an exception on None, we catch that and puts in np.nan). The exception handling should be relatively slower.

Timing seems to verify this hypothesis:

import numpy as np
%timeit [None] * 100000
> 1000 loops, best of 3: 1.04 ms per loop

%timeit np.array([0.0] * 100000)
> 10 loops, best of 3: 21.3 ms per loop
%timeit [i.__float__() for i in [0.0] * 100000]
> 10 loops, best of 3: 32 ms per loop


def flt(d):
    try:
        return float(d)
    except:
        return np.nan

%timeit np.array([None] * 100000, dtype=np.float64)
> 1 loops, best of 3: 477 ms per loop    
%timeit [flt(d) for d in [None] * 100000]
> 1 loops, best of 3: 328 ms per loop

Adding another case just to be obvious about where I'm going with this. If there was an explicit check for None, it would not be this slow above:

def flt2(d):                              
    if d is None:
        return np.nan
    try:
        return float(d)
    except:
        return np.nan

%timeit [flt2(d) for d in [None] * 100000]
> 10 loops, best of 3: 45 ms per loop