Saving List of Numpy 2D arrays using numpy.save (the arrays together are jagged)

Question

I have a large image dataset. When I use the images, I have several components--a mirrored image, a regular image, an eigenvector matrix and an eigenvalue vector.

I would like to store it like:

training_sunsets_data = [cropped_training_sunsets,
                         mirrored_training_sunsets,
                         rgb_cov_eigvec_training_sunsets,
                         rgb_cov_eigval_training_sunsets]

np.save('training_sunsets_data',training_sunsets_data)

And as I was writing this I was testing it (because I was sure it would fail), and the strangest thing happened when I did this: it worked.

Further, when I loaded it back up into the code, it was type ndarray, but it is a jagged array.

---

How is this possible if numpy does not allow jagged multidimensional arrays? Did I just find a backdoor way to create a jagged array in numpy?

Answer 1

After testing on my machine:

  import numpy as np
  np.save('testnp.npy', [[2,3,4],[1,2]])
  np.load('testnp.npy')
  #   array([[2, 3, 4], [1, 2]], dtype=object)

As shown in the example code, the loaded object is of type ndarray, but its data type is object. That means, np.save store an array of python objects, which can be anything. According to the documentation, it seems to use python pickle to pack those objects.

So you didn't find a backdoor, it behaves just as expected.

Answer 2

np.savez() would work in your situation. save each as a variable.