Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to flatten a memoryview?

Tags:

I have a memoryview with non-trivial strides like the following:

>>> mv.strides
(96, 32, 8)

I want to write this memoryview to a socket but my networking library seems to expect memoryviews with mv.strides == (1,). Is there a way in Python to flatten this memoryview?

>>> flatten(mv).strides
(1,)

Ideally this would neither affect the underlying bytes nor require a copy. I could do this with NumPy, but I'd rather keep things general if possible.

Edit: Here is some code that produces such a memoryview

In [1]: import numpy as np
In [2]: x = np.ones((2, 3, 4))
In [3]: x.data
Out[3]: <memory at 0x7f371aa849a8>

In [4]: x.data.strides
Out[4]: (96, 32, 8)
like image 833
MRocklin Avatar asked Jun 11 '17 16:06

MRocklin


People also ask

When should I use Memoryview?

memoryview objects are great when you need subsets of binary data that only need to support indexing. Instead of having to take slices (and create new, potentially large) objects to pass to another API you can just take a memoryview object. One such API example would be the struct module.

How do I use Memoryview?

The memoryview() function allows direct read and write access to an object's byte-oriented data without needing to copy it first. That can yield large performance gains when operating on large objects since it doesn't create a copy when slicing. Parameters: obj – object whose internal data is to be exposed.


1 Answers

Just for clarification, you probably know this but I think it's better to make sure:

  • The length of the strides tuple represents the number of dimensions, so (1, ) and (8, ) are both one dimensional and (10, 2) and (20, 1) are both two-dimensional.
  • For C-contiguous arrays the last element in the strides tuple represents the itemsize of the items in your memoryview. That's not always correct: sometimes the values are padded then it will be bigger than the actual itemsize - but in most cases it represents the itemsize.

So you don't only want your memoryview flattened but it should be flattened and have an itemsize of 1.

In Python 3.3 the memoryview.cast method was added that makes flattening your array trivial:

cast(format[, shape])

Cast a memoryview to a new format or shape. shape defaults to [byte_length//new_itemsize], which means that the result view will be one-dimensional. The return value is a new memoryview, but the buffer itself is not copied. Supported casts are 1D -> C-contiguous and C-contiguous -> 1D.

The destination format is restricted to a single element native format in struct syntax. One of the formats must be a byte format (‘B’, ‘b’ or ‘c’). The byte length of the result must be the same as the original length.

So it only works if you cast to char (c), unsigned char (B) or signed chars (b) and it's C-contiguous.

>>> import numpy as np

>>> memview = memoryview(np.ones((2, 3, 4)))
>>> memview.cast('b').strides   # or 'B' or 'c'
(1, )

However this is flattened and interpreted as 1-byte values. If you just want it flattened you need to cast it to the original type again:

>>> memview.cast('b').cast(memview.format)

That will be one-dimensional but it won't have strides of (1, ) because floats are 8 bytes (at least if it's float64):

>>> memview.cast('b').cast(memview.format).strides
(8, )
like image 179
MSeifert Avatar answered Oct 12 '22 21:10

MSeifert