Fastest way from logic matrix to list of sets

Question

I need to convert a sparse logic matrix into a list of sets, where each list[i] contains the set of rows with nonzero values for column[i]. The following code works, but I'm wondering if there's a faster way to do this. The actual data I'm using is approx 6000x6000 and much more sparse than this example.

import numpy as np

A = np.array([[1, 0, 0, 0, 0, 1],
              [0, 1, 1, 1, 1, 0],
              [1, 0, 1, 0, 1, 1],
              [1, 1, 0, 1, 0, 1],
              [1, 1, 0, 1, 0, 0],
              [1, 0, 0, 0, 0, 0],
              [0, 0, 1, 1, 1, 0],
              [0, 0, 1, 0, 1, 0]])

rows,cols = A.shape

C = np.nonzero(A)
D = [set() for j in range(cols)]

for i in range(len(C[0])):
    D[C[1][i]].add(C[0][i])

print D

Answer 1

If you represent the sparse array as a csc_matrix, you can use the indices and indptr attributes to create the sets.

For example,

In [93]: A
Out[93]: 
array([[1, 0, 0, 0, 0, 1],
       [0, 1, 1, 1, 1, 0],
       [1, 0, 1, 0, 1, 1],
       [1, 1, 0, 1, 0, 1],
       [1, 1, 0, 1, 0, 0],
       [1, 0, 0, 0, 0, 0],
       [0, 0, 1, 1, 1, 0],
       [0, 0, 1, 0, 1, 0]])

In [94]: from scipy.sparse import csc_matrix

In [95]: C = csc_matrix(A)

In [96]: C.indptr
Out[96]: array([ 0,  5,  8, 12, 16, 20, 23], dtype=int32)

In [97]: C.indices
Out[97]: array([0, 2, 3, 4, 5, 1, 3, 4, 1, 2, 6, 7, 1, 3, 4, 6, 1, 2, 6, 7, 0, 2, 3], dtype=int32)

In [98]: D = [set(C.indices[C.indptr[i]:C.indptr[i+1]]) for i in range(C.shape[1])]

In [99]: D
Out[99]: 
[{0, 2, 3, 4, 5},
 {1, 3, 4},
 {1, 2, 6, 7},
 {1, 3, 4, 6},
 {1, 2, 6, 7},
 {0, 2, 3}]

For a list of arrays instead of sets, just don't call set():

In [100]: [C.indices[C.indptr[i]:C.indptr[i+1]] for i in range(len(C.indptr)-1)]
Out[100]: 
[array([0, 2, 3, 4, 5], dtype=int32),
 array([1, 3, 4], dtype=int32),
 array([1, 2, 6, 7], dtype=int32),
 array([1, 3, 4, 6], dtype=int32),
 array([1, 2, 6, 7], dtype=int32),
 array([0, 2, 3], dtype=int32)]

Answer 2

Since you already called np.nonzero on A, see if this works faster:

>>> from itertools import groupby
>>> C = np.transpose(np.nonzero(A.T))
>>> [{i[1] for i in g} for _, g in groupby(C, key=lambda x: x[0])]
[{0, 2, 3, 4, 5}, {1, 3, 4}, {1, 2, 6, 7}, {1, 3, 4, 6}, {1, 2, 6, 7}, {0, 2, 3}]

Some timing:

In [4]: %%timeit
   ...: C = np.transpose(np.nonzero(A.T))
   ...: [{i[1] for i in g} for _, g in groupby(C, key=lambda x: x[0])]
   ...:
10000 loops, best of 3: 39 µs per loop

In [7]: %%timeit
   ...: C=csc_matrix(A)
   ...: [set(C.indices[C.indptr[i]:C.indptr[i+1]]) for i in range(C.shape[1])]
   ...:
1000 loops, best of 3: 317 µs per loop

Answer 3

I don't know if increases speed much, but your iteration can streamlined with

for i,j in zip(*C):
    D[j].add(i)

A defaultdict could add a nice touch to this task:

In [58]: from collections import defaultdict    
In [59]: D=defaultdict(set)
In [60]: for i,j in zip(*C):
    D[j].add(i)

In [61]: D
Out[61]: defaultdict(<class 'set'>, {0: {0, 2, 3, 4, 5}, 1: {1, 3, 4}, 2: {1, 2, 6, 7}, 3: {1, 3, 4, 6}, 4: {1, 2, 6, 7}, 5: {0, 2, 3}})

In [62]: dict(D)
Out[62]: 
{0: {0, 2, 3, 4, 5},
 1: {1, 3, 4},
 2: {1, 2, 6, 7},
 3: {1, 3, 4, 6},
 4: {1, 2, 6, 7},
 5: {0, 2, 3}}

An alternative with sparse matrices is the lil format which saves the data a list of lists. Since you want to collect data by column, make the matrix from A.T (transpose)

In [70]: M=sparse.lil_matrix(A.T)

In [71]: M.rows
Out[71]: 
array([[0, 2, 3, 4, 5], [1, 3, 4], [1, 2, 6, 7], [1, 3, 4, 6],
       [1, 2, 6, 7], [0, 2, 3]], dtype=object)

Which are the same lists.

For this small case direct iteration is faster than sparse

In [72]: %%timeit 
   ....: D=defaultdict(set)
   ....: for i,j in zip(*C):
    D[j].add(i)
   ....: 
10000 loops, best of 3: 24.4 µs per loop

In [73]: %%timeit
   ....: D=[set() for j in range(A.shape[1])]
   ....: for i,j in zip(*C):
    D[j].add(i)
   ....: 
10000 loops, best of 3: 22.9 µs per loop

In [74]: %%timeit 
   ....: M=sparse.lil_matrix(A.T)
   ....: M.rows
   ....: 
1000 loops, best of 3: 588 µs per loop

In [75]: %%timeit
   ....: C=sparse.csc_matrix(A)
   ....: D = [set(C.indices[C.indptr[i]:C.indptr[i+1]]) for i in range(C.shape[1])]   ....: 
1000 loops, best of 3: 476 µs per loop

For a large array, the setup time for the sparse matrix will less significant.

==========================

Do we really need set? A variation on the lil approach is to start with the nonzero on the transpose, i.e. by column

In [90]: C=np.nonzero(A.T)

# (array([0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5], dtype=int32),
# array([0, 2, 3, 4, 5, 1, 3, 4, 1, 2, 6, 7, 1, 3, 4, 6, 1, 2, 6, 7, 0, 2, 3], dtype=int32))

The numbers are all there; we just have to split the 2nd list into pieces corresponding to the first

In [91]: i=np.nonzero(np.diff(C[0]))[0]+1

In [92]: np.split(C[1],i)
Out[92]: 
[array([0, 2, 3, 4, 5], dtype=int32),
 array([1, 3, 4], dtype=int32),
 array([1, 2, 6, 7], dtype=int32),
 array([1, 3, 4, 6], dtype=int32),
 array([1, 2, 6, 7], dtype=int32),
 array([0, 2, 3], dtype=int32)]

This is slower than the direct iteration but I suspect it scales better; possibly as well as any of the sparse alternatives:

In [96]: %%timeit 
C=np.nonzero(A.T)
   ....: i=np.nonzero(np.diff(C[0]))[0]+1
   ....: np.split(C[1],i)
   ....: 
10000 loops, best of 3: 55.2 µs per loop