Send opencv Mat image through Qt websocket to HTML client

Question

I'm trying to write an application in c++ using Qt 5.7, basically it should be a websocket server, using qwebsocket for this, capable to send an image elaborated with opencv to an HTML client. What I'm trying to do is encode the image in base64, transmit and on the client put the encoded string in the src of an image tag. Just to test, I can send/receive text messages correctly, so the websocket architecture is working, but I have some problems with images. This is my code snippets:

Server

    cv::Mat imgIn;
    imgIn = cv::imread("/home/me/color.png",CV_LOAD_IMAGE_COLOR);
    QByteArray Img((char*)(imgIn.data),imgIn.total()*imgIn.elemSize());
    QByteArray Img64 = Img.toBase64();
    pClient->sendBinaryMessage(Img64);

Client

<img id="ItemPreview" src="" style="border:5px solid black" />

....

websocket.binaryType = "arraybuffer";

websocket.onmessage = function (evt) {
console.log( "Message received :", evt.data );
document.getElementById("ItemPreview").src = "data:image/png;base64," + evt.data;
};

I think most of the problems are in the Server, because the base64 sequence I got from the image is different from the one I can get from online converter image/base64. On the client I receive this error in the console and nothing is showed:

data:image/png;base64,[object ArrayBuffer]:1 GET data:image/png;base64,[object ArrayBuffer] net::ERR_INVALID_URL

Any hints?

SOLUTION

Thanks to the suggestions, I can provide the working code:

Server

    imgIn = cv::imread("/home/me/color.png", CV_LOAD_IMAGE_UNCHANGED);
    std::vector<uchar> buffer;
    cv::imencode(".png",imgIn,buffer);
    std::string s = base64_encode(buffer.data(),buffer.size());
    pClient->sendTextMessage(QString::fromStdString(s));

Client

Removed this line:

    websocket.binaryType = "arraybuffer";

The base64 encoding in the server is done using this code:

Encode/Decode base64

Answer 1

This line in the server:

imgIn = cv::imread("/home/me/color.png",CV_LOAD_IMAGE_COLOR);

decodes a PNG formatted image, and places it in memory as load of pixel data (plus possibly some row padding, which you don't take account of, see below). That's what you're base64 encoding.

This line in the client:

document.getElementById("ItemPreview").src = "data:image/png;base64," + evt.data;

is expecting a PNG image, but that isn't what you're sending; you've just pushed out a load of raw pixel data, with no dimensions or stride or format information or anything else.

If your client wants a PNG, you're going to have to use something like imencode to write PNG data to a memory buffer, and base64 encode that instead.

One other important thing to note is that decoded images may have row padding... a few bytes on the end of each row for memory alignment purposes. Therefore, the actual length of each image row may exceed the width of the image multiplied by the size of the each pixel in bytes. That means that this operation:

QByteArray Img((char*)(imgIn.data),imgIn.total()*imgIn.elemSize());

may not, in fact, wrap the entire image buffer in your QByteArray. There are various ways to check the stride/step of an image, but you'd best read the cv::Mat docs as it isn't worth me repeating them all here. This only matters if you're doing raw byte-level image manipulation, like you are here. If you use imencode, you don't need to worry about this.