Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

cython: memory view of ndarray of strings (or direct ndarray indexing)

How does one specify a memory view of a ndarray that contains strings?

char[:], char*[:], ... do not work.

To illustrate, my problem is the definition of function abc(...):

cdef void abc(char[:] in_buffer):
    cdef char * element
    element = address(in_buffer[1])
    ...

def main():
    cdef Py_ssize_t i, n = 100

    a = np.array(['ABC', 'D', 'EFGHI'])
    for i in range(n):
        abc(a)

If a memoryview is not possible, can I implement direct array access myself? I need to avoid the GIL for function abc(...).


Edit 1: In response to Bi Rico's answer.

My aim is to release the GIL for function abc(...) and within it process the string elements of ndarray in_buffer with c string-functions. I.e. something like the following:

cdef void abc(char[:, ::1] in_buffer) nogil:
    cdef int max_elt_length = in_buffer.shape[1]+1
    cdef char element[max_elt_length+1]
    cdef int length

    for i in range(in_buffer.shape[0]+1):  # is this equivalent to in_buffer.dtype.itemsize + 1 ?
       element[max_elt_length] = 0   # add null-terminator for full-size elements
       memcpy(element, address(buffer[i, 0]), max_length)
       length = strlen(element)
       ...
like image 277
ARF Avatar asked Feb 27 '15 20:02

ARF


1 Answers

The issue is that numpy array dtypes have to have a fixed size. When you make an array of "strings" you're actually making an array of fixed length char arrays. Try this:

import numpy as np

array = np.array(["cat", "in", "a", "hat"])
array[2] = "Seuss"
print(array)
# ['cat' 'in' 'Seu' 'hat']
print(array.dtype)
# dtype('|S3')
print(array.dtype.itemsize)
# 3

With that in mind, you could something like this:

cdef void abc(char[:, ::1] in_buffer):
    cdef char * element
    element = address(in_buffer[1, 0])

Then when you pass your arrays to abc you'll need to do something like:

a = np.array(['ABC', 'D', 'EFGHI'])
array_view = a.view('uint8').reshape(a.size, a.dtype.itemsize)
abc(array_view)

This is only one approach, but it's the one I would recommend without knowing more about what you're trying to do.

like image 143
Bi Rico Avatar answered Sep 21 '22 01:09

Bi Rico