Getting numpy error (ValueError: setting an array element with a sequence)

Question

The following code written using python 3.6. It should create a final matrix containing binary vectors. During the loop, each batch is taken from consecutive vectors to be used by a method simulate_output() but eventually the batch is flatten to be a single vector:

tmp_list = []
output = []
num_components = 4
dim = 16
total_vectors = 100000

X = np.random.choice(np.array([0, 1], dtype=np.uint8), size=(total_vectors, dim))
num_vectors = np.unique(X, axis=0)

for i in range(0, len(num_vectors), num_components):
    batch = num_vectors[i:(i + num_components)]
    # output.append(simulate_output(batch)) # Comment this line will not solve the error.
    batch = np.hstack(batch)  # to flatten the list into a single vector
    tmp_list.append(batch)

final_matrix = np.array(tmp_list, dtype=np.int8)
print(final_matrix)

For some runs I get this error:

Traceback (most recent call last):
  File "test.py", line 65, in <module>
    final_matrix = np.array(tmp_list, dtype=np.int8)
ValueError: setting an array element with a sequence.

I believe the error is on last line final_matrix = np.array(tmp_list, dtype=np.int8) but I have no idea why and how to fix it since in some runs it works while in other runs it does not.

Thank you

Answer 1

I found your problem. In this line:

final_matrix = np.array(tmp_list, dtype=np.int8)

you expect the final_matrix to be a 2-dimensional numpy array. this can be if all the lines have the same length, but this is not exactly your case. your last flatten batch vector is shorter because len(num_vectors) is not divided by num_components(4).

If you simply put:

tmp_list = tmp_list[:-1]

after the for loop, everything will be fine. I think one element out of thousands is negligible. If you still don't want to delete it, try to pad it with zeroes to the desired size - num_components * dim.