Named dtype array: Difference between a[0]['name'] and a['name'][0]?

Question

I came across the following oddity in numpy which may or may not be a bug:

import numpy as np
dt = np.dtype([('tuple', (int, 2))])
a = np.zeros(3, dt)
type(a['tuple'][0])  # ndarray
type(a[0]['tuple'])  # ndarray

a['tuple'][0] = (1,2)  # ok
a[0]['tuple'] = (1,2)  # ValueError: shape-mismatch on array construction

I would have expected that both of the options below work. Opinions?

Answer 1

I asked that on the numpy-discussion list. Travis Oliphant answered here.

Citing his answer:

The short answer is that this is not really a "normal" bug, but it could be considered a "design" bug (although the issues may not be straightforward to resolve). What that means is that it may not be changed in the short term --- and you should just use the first spelling.

Structured arrays can be a confusing area of NumPy for several of reasons. You've constructed an example that touches on several of them. You have a data-type that is a "structure" array with one member ("tuple"). That member contains a 2-vector of integers.

First of all, it is important to remember that with Python, doing

a['tuple'][0] = (1,2)

is equivalent to

b = a['tuple']; b[0] = (1,2)

In like manner,

a[0]['tuple'] = (1,2)

is equivalent to

b = a[0]; b['tuple'] = (1,2)

To understand the behavior, we need to dissect both code paths and what happens. You built a (3,) array of those elements in 'a'. When you write b = a['tuple'] you should probably be getting a (3,) array of (2,)-integers, but as there is currently no formal dtype support for (n,)-integers as a general dtype in NumPy, you get back a (3,2) array of integers which is the closest thing that NumPy can give you. Setting the [0] row of this object via

a['tuple'][0] = (1,2)

works just fine and does what you would expect.

On the other hand, when you type:

b = a[0]

you are getting back an array-scalar which is a particularly interesting kind of array scalar that can hold records. This new object is formally of type numpy.void and it holds a "scalar representation" of anything that fits under the "VOID" basic dtype.

For some reason:

b['tuple'] = [1,2]

is not working. On my system I'm getting a different error: TypeError: object of type 'int' has no len()

I think this should be filed as a bug on the issue tracker which is for the time being here: http://projects.scipy.org/numpy

The problem is ultimately the void->copyswap function being called in voidtype_setfields if someone wants to investigate. I think this behavior should work.

An explanation for this is given in a numpy bug report.

Answer 2

I get a different error than you do (using numpy 1.7.0.dev):

ValueError: setting an array element with a sequence.

so the explanation below may not be correct for your system (or it could even be the wrong explanation for what I see).

First, notice that indexing a row of a structured array gives you a numpy.void object (see data type docs)

import numpy as np
dt = np.dtype([('tuple', (int, 2))])
a = np.zeros(3, dt)
print type(a[0]) # = numpy.void

From what I understand, void is sort of like a Python list since it can hold objects of different data types, which makes sense since the columns in a structured array can be different data types.

If, instead of indexing, you slice out the first row, you get an ndarray:

print type(a[:1]) # = numpy.ndarray

This is analogous to how Python lists work:

b = [1, 2, 3]
print b[0] # 1
print b[:1] # [1]

Slicing returns a shortened version of the original sequence, but indexing returns an element (here, an int; above, a void type).

So when you slice into the rows of the structured array, you should expect it to behave just like your original array (only with fewer rows). Continuing with your example, you can now assign to the 'tuple' columns of the first row:

a[:1]['tuple'] = (1, 2)

So,... why doesn't a[0]['tuple'] = (1, 2) work?

Well, recall that a[0] returns a void object. So, when you call

a[0]['tuple'] = (1, 2) # this line fails

you're assigning a tuple to the 'tuple' element of that void object. Note: despite the fact you've called this index 'tuple', it was stored as an ndarray:

print type(a[0]['tuple']) # = numpy.ndarray

So, this means the tuple needs to be cast into an ndarray. But, the void object can't cast assignments (this is just a guess) because it can contain arbitrary data types so it doesn't know what type to cast to. To get around this you can cast the input yourself:

a[0]['tuple'] = np.array((1, 2))

The fact that we get different errors suggests that the above line might not work for you since casting addresses the error I received---not the one you received.

Addendum:

So why does the following work?

a[0]['tuple'][:] = (1, 2)

Here, you're indexing into the array when you add [:], but without that, you're indexing into the void object. In other words, a[0]['tuple'][:] says "replace the elements of the stored array" (which is handled by the array), a[0]['tuple'] says "replace the stored array" (which is handled by void).

Epilogue:

Strangely enough, accessing the row (i.e. indexing with 0) seems to drop the base array, but it still allows you to assign to the base array.

print a['tuple'].base is a # = True
print a[0].base is a # = False
a[0] = ((1, 2),) # `a` is changed

Maybe void is not really an array so it doesn't have a base array,... but then why does it have a base attribute?