msgpack C++ implementation: How to pack binary data?

Question

I am making use of C++ msgpack implementation. I have hit a roadblock as to how to pack binary data. In terms of binary data I have a buffer of the following type:

unsigned char* data;

The data variable points to an array which is actually an image. What I want to do is pack this using msgpack. There seems to be no example of how to actually pack binary data. From the format specification raw bytes are supported, but I am not sure how to make use of the functionality.

I tried using a vector of character pointers like the following:

msgpack::sbuffer temp_sbuffer;
std::vector<char*> vec;
msgpack::pack(temp_sbuffer, vec);

But this results in a compiler error since there is no function template for T=std::vector.

I have also simply tried the following:

msgpack::pack(temp_sbuffer, "Hello");

But this also results in a compilation error (i.e. no function template for T=const char [6]

Thus, I was hoping someone could give me advice on how to use msgpack C++ to pack binary data represented as a char array.

Answer 1

Josh provided a good answer but it requires the copying of byte buffers to a vector of char. I would rather minimize copying and use the buffer directly (if possible). The following is an alternative solution:

Looking through the source code and trying to determine how different data types are packed according to the specification I happened upon msgpack::packer<>::pack_raw(size_t l) and msgpack::packer<>::pack_raw_body(const char* b, size_t l). While there appears to be no documentation for these methods this is how I would described them.

1. msgpack::packer<>::pack_raw(size_t l): This method appends the type identification to buffer (i.e. fix raw, raw16 or raw32) as well as the size information (which is an argument for the method).
2. msgpack::packer<>::pack_raw_body(const char* b, size_t l): This method appends the raw data to the buffer.

The following is a simple example of how to pack a character array:

msgpack::sbuffer temp_sbuffer;
msgpack::packer<msgpack::sbuffer> packer(&temp_sbuffer);
packer.pack_raw(5);  // Indicate that you are packing 5 raw bytes
packer.pack_raw_body("Hello", 5); // Pack the 5 bytes

The above example can be extended to pack any binary data. This allows one to pack directly from byte arrays/buffers without having to copy to an intermediate (i.e. a vector of char).

Answer 2

If you can store your image in a vector<unsigned char> instead of a raw array of unsigned char, then you can pack that vector:

#include <iostream>
#include <string>
#include <vector>
#include <msgpack.hpp>

int main()
{
    std::vector<unsigned char> data;
    for (unsigned i = 0; i < 10; ++i)
        data.push_back(i * 2);

    msgpack::sbuffer sbuf;
    msgpack::pack(sbuf, data);

    msgpack::unpacked msg;
    msgpack::unpack(&msg, sbuf.data(), sbuf.size());

    msgpack::object obj = msg.get();
    std::cout << obj << std::endl;
}

Strangely, this only works for unsigned char. If you try to pack a buffer of char instead (or even an individual char), it won't compile.

Answer 3

MessagePack has a raw_ref type which you could use like so:

#include "msgpack.hpp"

class myClass
{
public:
    msgpack::type::raw_ref r;
    MSGPACK_DEFINE(r);
};

int _tmain(int argc, _TCHAR* argv[])
{
    const char* str = "hello";

    myClass c;

    c.r.ptr = str;
    c.r.size = 6;

    // From here on down its just the standard MessagePack example...

    msgpack::sbuffer sbuf;
    msgpack::pack(sbuf, c);

    msgpack::unpacked msg;
    msgpack::unpack(&msg, sbuf.data(), sbuf.size());

    msgpack::object o = msg.get();

    myClass d;
    o.convert(&d);

    OutputDebugStringA(d.r.ptr);

    return 0;

}

Disclaimer: I found this by poking around the header files, not through reading the non-existent documentation on serialising raw bytes, so it may not be the 'correct' way (though it was defined along with all the other 'standard' types a serialiser would want to explicitly handle).

Answer 4

msgpack-c has been updated after question and answers were posted. I'd like to inform the current situation.

Since msgpack-c version 2.0.0 C-style array has been supported. See https://github.com/msgpack/msgpack-c/releases

msgpack-c can pack const char array such as "hello". Types conversion rule is documented https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor#predefined-adaptors.

char array is mapped to STR. If you want to use BIN instead of STR, you need to wrap with msgpack::type::raw_ref. That is packing overview.

Here are unpacking and converting description: https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_object#conversion

Unpack means creating msgpack::object from MessagePack formatted byte stream. Convert means converting to C++ object from msgpack::object.

If MessagePack formatted data is STR, and covert target type is char array, copy data to array, and if array has extra capacity, add '\0'. If MessagePack formatted data is BIN, '\0' is not added.

Here is a code example based on the original question:

#include <msgpack.hpp>
#include <iostream>

inline
std::ostream& hex_dump(std::ostream& o, char const* p, std::size_t size ) {
    o << std::hex << std::setw(2) << std::setfill('0');
    while(size--) o << (static_cast<int>(*p++) & 0xff) << ' ';
    return o;
}

int main() {
    {
        msgpack::sbuffer temp_sbuffer;
        // since 2.0.0 char[] is supported.
        // See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor#predefined-adaptors
        msgpack::pack(temp_sbuffer, "hello");
        hex_dump(std::cout, temp_sbuffer.data(), temp_sbuffer.size()) << std::endl;

        // packed as STR See https://github.com/msgpack/msgpack/blob/master/spec.md
        // '\0' is not packed
        auto oh = msgpack::unpack(temp_sbuffer.data(), temp_sbuffer.size());
        static_assert(sizeof("hello") == 6, "");
        char converted[6];
        converted[5] = 'x'; // to check overwriting, put NOT '\0'.
        // '\0' is automatically added if char-array has enought size and MessagePack format is STR
        oh.get().convert(converted); 
        std::cout << converted << std::endl;
    }
    {
        msgpack::sbuffer temp_sbuffer;
        // since 2.0.0 char[] is supported.
        // See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor#predefined-adaptors
        // packed as BIN
        msgpack::pack(temp_sbuffer, msgpack::type::raw_ref("hello", 5));
        hex_dump(std::cout, temp_sbuffer.data(), temp_sbuffer.size()) << std::endl;

        auto oh = msgpack::unpack(temp_sbuffer.data(), temp_sbuffer.size());
        static_assert(sizeof("hello") == 6, "");
        char converted[7];
        converted[5] = 'x';
        converted[6] = '\0';
        // only first 5 bytes are written if MessagePack format is BIN
        oh.get().convert(converted);
        std::cout << converted << std::endl;
    }
}

Running Demo: https://wandbox.org/permlink/mYJyYycfsQIwsekY