Understanding PyTorch Tensor Shape

Question

I have a simple question regarding the shape of tensor we define in PyTorch. Let's say if I say:

input = torch.randn(32, 35)

This will create a matrix with 32 row and 35 columns. Now when I define:

input2 = torch.randn(1,2,32, 35)

What can I say about the dimension of the new matrix input2? How can I define the rows and columns here? I mean do I have two matrices with shapes 32*35 packed by the tensor?

I want to better understand the geometry behind this. Thanks.

Answer 1

Consider tensor shapes as the number of lists that a dimension holds. For instance, a tensor shaped (4, 4, 2) will have four elements, which will all contain 4 elements, which in turn have 2 elements.

1. The first holds 4 elements.
2. The second holds 4 elements.
3. The third dimension holds 2 elements.

Here's what the data would look like:

[[[0.86471446, 0.26302726],
  [0.04137454, 0.00349315],   
  [0.06559607, 0.45617865],
  [0.0219786, 0.27513594]],

 [[0.60555118, 0.10853228],
  [0.07059685, 0.32746256],
  [0.99684617, 0.07496456],
  [0.55169005, 0.39024103]],

 [[0.55891377, 0.41151245],
  [0.3434965, 0.12956237],
  [0.74908291, 0.69889266],
  [0.98600141, 0.8570597]],

 [[0.7903229, 0.93017741],
  [0.54663242, 0.72318166],
  [0.6099451, 0.96090241],
  [0.63772238, 0.78605599]]]

In other words, four elements of four elements of two elements.

Answer 2

Yes, that is correct. Your input2 tensor has a rank of 4. (Rank is the Dimension) and the bounds of each dimension are (1,2,32,35)

1. The first dimension can hold one element.
2. The second can hold two.
3. The third can hold 32 elements.
4. The forth dimension can hold 35 elements.

EDIT: I find it is useful to think of higher-dimensional arrays as a series of lists. In your case, a rank 4 tensor, would be a list of lists of lists of lists.