numpy append_field gives shape error for new field with 2d shape

Question

I have a structured numpy array, I want to use the recfunctions library http://pyopengl.sourceforge.net/pydoc/numpy.lib.recfunctions.html function append_fields() or rec_append_fields() to append a field with some shape to it. However, I get an error:

ValueError: operands could not be broadcast together with shapes (10) (10,3)

where 10 is the length of my existing array, and (3,) is the shape of the field I want to append.

For example:

import numpy as np
from numpy.lib.recfunctions import append_fields


my_structured_array = np.array(
    zip([0,1,2,3],[[4.3,3.2],[1.4,5.6],[6.,2.5],[4.5,5.4]]),
    dtype=[('id','int8'),('pos','2float16')]
    )
my_new_field = np.ones(
    len(my_structured_array),
    dtype='2int8'
    )
my_appended_array = append_fields(
    my_structured_array,
    'new',
    data=my_new_field
    )

ValueError: operands could not be broadcast together with shapes (4) (4,2)

Any ideas? I tried making my_new_field a list of tuples and putting a dtype argument with the proper shape into the append_fields():

my_new_field = len(my_structured_array)*[(1,1)]

my_appended_array = append_fields(
    my_structured_array,
    'new',
    data=my_new_field,
    dtype='2int8'
    )

but that seems to end up the same once it gets converted to a numpy array.

None of this seems to change when I use rec_append_fields() instead of simply append_fields()

EDIT: In light of the fact that my new field doesn't have the same shape as my array, I suppose that my desired append is impossible, suggested by @radicalbiscuit.

In : my_new_field.shape
Out: (4, 2)

In : my_structured_array.shape
Out: (4,)

But, I included one of the original fields in the array with shape different from the original array to make my point, which is that a field does not have to have the same shape as the structured array. How can I append a field like this?

In : my_structured_array['pos'].shape
Out: (4, 2)

In : my_new_field.shape
Out: (4, 2)

I should note that for my application, I can append an empty field as long as it's possible to somehow change the shape later. Thanks!

Answer 1

append_fields() does indeed require that the two arrays be the same shape. That being said, as you realized in my_structured_array, numpy does support subarrays (that is, a field can itself be an array with a shape).

In your case, I think you probably want my_new_field not to be a two dimensional array, but instead be a one dimensional array (of shape shape(my_structured_array)) with elements of dtype, e.g., dtype([('myfield', '<i8', (2,))]). For example,

import numpy as np
from numpy.lib.recfunctions import append_fields

my_structured_array = np.array(
    zip([0,1,2,3],[[4.3,3.2],[1.4,5.6],[6.,2.5],[4.5,5.4]]),
    dtype=[('id','int8'),('pos','2float16')]
    )

my_new_field = np.ones(
    len(my_structured_array),
    dtype=[('myfield', 'i8', 2)]
    )

my_appended_array = append_fields(
    my_structured_array,
    'new',
    data=my_new_field
    )

Will yield,

>>> my_appended_array[0]
(0, [4.30078125, 3.19921875], ([1, 1],))

Although the datatype is slightly inconvenient as myfield is nested within new,

>>> my_appended_array.dtype
dtype([('id', '|i1'), ('pos', '<f2', (2,)), ('new', [('myfield', '<i8', (2,))])])

This, however, is coerced away fairly easily,

>>> np.asarray(my_appended_array, dtype=[('id', '|i1'), ('pos', '<f2', (2,)), ('myfield', '<i8', (2,))])
array([(0, [4.30078125, 3.19921875], [0, 0]),
       (1, [1.400390625, 5.6015625], [0, 0]), (2, [6.0, 2.5], [0, 0]),
       (3, [4.5, 5.3984375], [0, 0])], 
      dtype=[('id', '|i1'), ('pos', '<f2', (2,)), ('myfield', '<i8', (2,))])

Still, it's a bit unfortunate that we've had to repeat the dtype of my_structured_array here. While at first glance it appears that numpy.lib.recfunctions.flatten_descr could do the dirty work of flattening the dtype, it unfortunately gives a tuple and not a list as required by np.dtype. Coercing its output to a list, however, works around this issue,

>>> np.dtype(list(np.lib.recfunctions.flatten_descr(my_appended_array.dtype)))
dtype([('id', '|i1'), ('pos', '<f2', (2,)), ('myfield', '<i8', (2,))])

This can be passed as the dtype to np.asarray, making things slightly more robust against changes in my_structured_array.dtype.

Indeed, minor inconsistencies such as this make working with record arrays messy business. One gets the feeling that things could fit together a bit more coherently.

Edit: It turns out that the np.lib.recfunctions.merge_arrays function is much more amenable to this sort of merging,

 >>> my_appended_array = merge_arrays([my_structured_array, my_new_field], flatten=True)
 array([(0, [4.30078125, 3.19921875], [1, 1]),
        (1, [1.400390625, 5.6015625], [1, 1]), (2, [6.0, 2.5], [1, 1]),
        (3, [4.5, 5.3984375], [1, 1])], 
       dtype=[('id', '|i1'), ('pos', '<f2', (2,)), ('myfield', '<i8', (2,))])

Answer 2

append_fields() requires that the two arrays are the same shape, which in this case they are not. Printing out the two arrays will help it become obvious:

>>> my_structured_array
array([(0, [4.30078125, 3.19921875]), (1, [1.400390625, 5.6015625]),
       (2, [6.0, 2.5]), (3, [4.5, 5.3984375])], 
      dtype=[('id', '|i1'), ('pos', '<f2', (2,))])
>>> my_new_field
array([[1, 1],
       [1, 1],
       [1, 1],
       [1, 1]], dtype=int8)

As you can see, my_structured_array is an array of length 4 where each element is a tuple containing two objects, an int and a list of two floats.

my_new_field, on the other hand, is an array of length 4 where each element is a list of two ints. It's like trying to add apples and oranges.

Make your arrays the same shape and they'll add together.