numpy ndarray with more that 32 dimensions

Question

When I try to create an numpy array with more than 32 dimensions I get an error:

import numpy as np

np.ndarray([1] * 33)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-78103e601d91> in <module>()
----> 1 np.ndarray([1] * 33)

ValueError: sequence too large; cannot be greater than 32

I found this: Using numpy.array with large number of dimensions to be related to this question but I want to do this without building my own version.

My use case:
I am working with Joint Probability Distributions and I am trying to represent each variable on an axis so that computations on it (marginalize, reduce) is a single line operation. For example for a marginalize operation I can simply do a sum over the axis of that variable. For multiplication I can simply do a simple numpy multiplication (after checking if the axes are the same).

Is there a possible workaround this?

Answer 1

Easy workaround

If you create a np.array(np.array(li))
Where li is a list and len(li) > 32, it'll return an ndarray as intended.

Answer 2

How about maintaining the data in a 1d, and selectively reshaping to focus on a given dimension

To illustrate

In [252]: x=np.arange(2*3*4*5)

With full reshape

In [254]: x.reshape(2,3,4,5).sum(2)
Out[254]: 
array([[[ 30,  34,  38,  42,  46],
        [110, 114, 118, 122, 126],
        [190, 194, 198, 202, 206]],

       [[270, 274, 278, 282, 286],
        [350, 354, 358, 362, 366],
        [430, 434, 438, 442, 446]]])

With partial reshape - same numbers, different result shape

In [255]: x.reshape(6,4,5).sum(1)
Out[255]: 
array([[ 30,  34,  38,  42,  46],
       [110, 114, 118, 122, 126],
       [190, 194, 198, 202, 206],
       [270, 274, 278, 282, 286],
       [350, 354, 358, 362, 366],
       [430, 434, 438, 442, 446]])

I'm not going to test this on anything approaching 32+ dimensions. As was noted in the comment, if many of the dimensions are larger than 1 the overall array size will be excessively large.

Answer 3

As of 11/17/2020, the limit to the number of dimensions for numpy arrays remains 32. To find out I ran the following code:

for dim in range (1, 100):
    arr_n_dim_list = [1]*dim
    arr_n_dim = np.ones((arr_n_dim_list))
    print(arr_n_dim.shape)

The last line of the outputs is:

ValueError: maximum supported dimension for an ndarray is 32, found 33