Send numpy array as bytes from python to JS through Flask

Question

A numpy array should be sent from python to JS through flask. I don't want to jsonify it and send as it would increase the size of the response and eventually the response time. So I thought I could convert numpy into bytes using tobytes(), send the bytes to JS through flask and convert the bytes back to float in JS and found Converting a string of packed bytes into an array of floats in Javascript answer helpful.

The list of operations are

byte_arr = np.array([5.6], dtype=np.float32).tobytes()  # Used one value just to make it simple
return byte_arr  # Bytes can be directly sent from flask as response

In JS,

str = response.text
bytes = Uint8Array.from(str, c => c.charCodeAt(0))
floats = new Float32Array(bytes.buffer)

But when converting to float in JS, I am not getting the correct values. When debugging I found that in JS some values in Uint8 array are not matching with the byte values that are sent from python.

JS: console.log(bytes)  // [51, 51, 255, 64]
Python: for val in byte_arr: print(val, end=" ")  // 51, 51, 179, 64

Just to verify, I created another client in python, made the same request and converted the response bytes to float. I encountered the same issue here also. So I narrowed down that the problem is on the server side not on the client side. One more interesting thing I found when playing with multiple values are, only the numbers which are greater than 127 (I guess so) are getting converted to 255.

While trying some random hacks, I tried to convert the byte integer to character using chr() method of python before sending the response and the conversion on the client side worked.

byte_arr = np.array([5.6], dtype=np.float32).tobytes()
byte_arr_char = "".join([chr(i) for i in byte_arr])
return byte_arr_char

But the question I have is, Is this an ideal solution or am I doing some hack here to make it work? Can anyone help me to understand why sending plain bytes without chr() doesn't work?

Answer 1

If you don't set the MIME type explicitly, I think Flask will treat it as text data. Your browser seems to have decoded the binary data with ASCII, which can explain why only values larger than 127 were affected.

Therefore, please try setting the Content-Type of the response in Flask:

@app.route('/your/url/to/numpy/data')
def get_nparray():
    your_np_array = np.array([5.6], dtype=np.float32)
    response = flask.make_response(your_np_array.tobytes())
    response.headers.set('Content-Type', 'application/octet-stream')
    # response.headers.set('Content-Disposition', 'attachment', filename='np-array.bin')
    return response

Alternatively, there is a helper function flask.send_file to construct the response in a single line. Please find an example here.

Besides this error, also pay attention to the endianness of your binary data, which is hardware specific. I would refer you to this answer (Javascript Typed Arrays and Endianness).