List of List to ndarray

Question

I am trying to use kmeans clustering in scipy, exactly the one present here:

http://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.vq.kmeans.html#scipy.cluster.vq.kmeans

What I am trying to do is to convert a list of list such as the following:

data without_x[
[0, 0, 0, 0, 0, 0, 0, 20.0, 1.0, 48.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1224.0, 125.5, 3156.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 22.5, 56.0, 41.5, 85.5, 0, 0, 0, 0, 0, 0, 0, 0, 1495.0, 3496.5, 2715.0, 5566.5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
]

into a ndarry in order to use it with the Kmeans method. When I try to convert the list of list into the ndarray I get an empty array, thus voiding the whole analysis. The length of the ndarray is variable and it depends on the number of samples gathered. But I can get that easily with the len(data_without_x)

Here is a snippet of the code that returns the empty list.

import numpy as np
import "other functions"

data, data_without_x = data_preparation.generate_sampled_pdf()
nodes_stats, k, list_of_list= result_som.get_number_k()

data_array = np.array(data_without_x)
whitened = whiten(data_array)
centroids, distortion = kmeans(whitened, int(k), iter=100000)

and this is what I get as output just saving in a simple log file:

___________________________
this is the data array[[ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 ..., 
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]]
___________________________
This is the whitened array[[ nan  nan  nan ...,  nan  nan  nan]
 [ nan  nan  nan ...,  nan  nan  nan]
 [ nan  nan  nan ...,  nan  nan  nan]
 ..., 
 [ nan  nan  nan ...,  nan  nan  nan]
 [ nan  nan  nan ...,  nan  nan  nan]
 [ nan  nan  nan ...,  nan  nan  nan]]
___________________________

Does anybody have a clue about what happens when I try to convert the list of list into a numpy.array?

Thanks for your help

Answer 1

That is exactly how to convert a list of lists to an ndarray in python. Are you sure your data_without_x is filled correctly? On my machine:

data = [[1,2,3,4],[5,6,7,8]]
data_arr = np.array(data)

data_arr
array([[1,2,3,4],
       [5,6,7,8]])

Which is the behavior I think you're expecting

Looking at your input you have a lot of zeros...keep in mind that the print out doesn't show all of it. You may just be seeing all the "zeros" from your input. Examine a specific non zero element to be sure