creating callbacks and structs for repeated field in a protobuf message in nanopb in c

Question

I have a proto message defined as:

message SimpleMessage {
repeated int32 number = 1;}

now, after compiling, the field is of pb_callback_t and I suppose to write that function. (without .options file)

now, where and what should the function contain? where does the data itself being stored and how can I access it/ assign new data to it?

* EDIT *

according to @Groo 's answer, this is the code I tried:

typedef struct {
    int numbers_decoded;
} DecodingState;

bool read_single_number(pb_istream_t *istream, const pb_field_t *field, void **arg)
{
    // get the pointer to the custom state
    DecodingState *state = (DecodingState*)(*arg);

    int32_t value;
    if (!pb_decode_varint32(istream, &value))
    {
        const char * error = PB_GET_ERROR(istream);
        printf("Protobuf error: %s", error);
        return false;
    }

    printf("Decoded successfully: %d", value);
    state->numbers_decoded++;

    return true;
}

int main(void) {
    int32_t arr[3] = {10, 22, 342};
    uint8_t buffer[128];
    size_t message_length;
    bool status;
    SimpleMessage simple = SimpleMessage_init_zero;

    printf("\nbefore : arr[0] = %d\n",arr[0]);

    // set the argument and the callback fn
    simple.number.arg = &arr;
    simple.number.funcs.decode = read_single_number;

    pb_ostream_t ostream = pb_ostream_from_buffer(buffer, sizeof(buffer));
    status = pb_encode(&ostream, SimpleMessage_fields, &simple);

    message_length = ostream.bytes_written;
    SimpleMessage simple1 = SimpleMessage_init_zero;
    simple = simple1;
    arr[0] = 0;
    pb_istream_t istream = pb_istream_from_buffer(buffer, message_length);
    // this function will call read_single_number several times
    status = pb_decode(&istream, SimpleMessage_fields, &simple);
    printf("\nafter : arr[0] = %d\n",arr[0]);

    return EXIT_SUCCESS;
}

and the output is:

before : arr[0] = 10

Decoded successfully: 17

after : arr[0] = 0

what do I do wrong?

Answer 1

You can use some nanopb-specific proto flags to force nanopb to generate structs with statically allocated arrays.

However, the default behavior of nanopb's protogen is to generate a callback function which is called by nanopb during encoding (once for the entire list) and decoding (once for each item in the list). This is sometimes preferred in low-memory embedded systems, because you don't need to allocate more than one item at a time.

So, for your .proto file:

message SimpleMessage {
    repeated int32 number = 1;
}

You might get something like:

typedef struct _SimpleMessage {
    pb_callback_t number;
} SimpleMessage;

Meaning you will have to create your own callback function which will be called for each item in succession.

So for simplicity, let's say you have a simple "variable length" list like this:

#define MAX_NUMBERS 32

typedef struct
{
    int32_t numbers[MAX_NUMBERS];
    int32_t numbers_count;
}
IntList;

// add a number to the int list
void IntList_add_number(IntList * list, int32_t number)
{
    if (list->numbers_count < MAX_NUMBERS)
    {
        list->numbers[list->numbers_count] = number;
        list->numbers_count++;
    }
}

Obviously, for such an example, using callbacks wouldn't make any sense, but it makes the example simple.

Encoding callback must iterate through the list, and write the protobuf tag and the value for each item in the list:

bool SimpleMessage_encode_numbers(pb_ostream_t *ostream, const pb_field_t *field, void * const *arg)
{
    IntList * source = (IntList*)(*arg);

    // encode all numbers
    for (int i = 0; i < source->numbers_count; i++)
    {
        if (!pb_encode_tag_for_field(ostream, field))
        {
            const char * error = PB_GET_ERROR(ostream);
            printf("SimpleMessage_encode_numbers error: %s", error);
            return false;
        }

        if (!pb_encode_svarint(ostream, source->numbers[i]))
        {
            const char * error = PB_GET_ERROR(ostream);
            printf("SimpleMessage_encode_numbers error: %s", error);
            return false;
        }
    }

    return true;
}

Decoding callback is called once for each item, and "appends" to the list:

bool SimpleMessage_decode_single_number(pb_istream_t *istream, const pb_field_t *field, void **arg)
{
    IntList * dest = (IntList*)(*arg);

    // decode single number
    int64_t number;
    if (!pb_decode_svarint(istream, &number))
    {
        const char * error = PB_GET_ERROR(istream);
        printf("SimpleMessage_decode_single_number error: %s", error);
        return false;
    }

    // add to destination list
    IntList_add_number(dest, (int32_t)number);
    return true;
}

With these two in place, you must be careful to assign the right callback to the right function:

uint8_t buffer[128];
size_t total_bytes_encoded = 0;

// encoding
{
    // prepare the actual "variable" array
    IntList actualData = { 0 };
    IntList_add_number(&actualData, 123);
    IntList_add_number(&actualData, 456);
    IntList_add_number(&actualData, 789);

    // prepare the nanopb ENCODING callback
    SimpleMessage msg = SimpleMessage_init_zero;
    msg.number.arg = &actualData;
    msg.number.funcs.encode = SimpleMessage_encode_numbers;

    // call nanopb
    pb_ostream_t ostream = pb_ostream_from_buffer(buffer, sizeof(buffer));
    if (!pb_encode(&ostream, SimpleMessage_fields, &msg))
    {
        const char * error = PB_GET_ERROR(&ostream);
        printf("pb_encode error: %s", error);
        return;
    }

    total_bytes_encoded = ostream.bytes_written;
    printf("Encoded size: %d", total_bytes_encoded);
}

And similar for decoding:

// decoding
{
    // empty array for decoding
    IntList decodedData = { 0 };

    // prepare the nanopb DECODING callback
    SimpleMessage msg = SimpleMessage_init_zero;
    msg.number.arg = &decodedData;
    msg.number.funcs.decode = SimpleMessage_decode_single_number;

    // call nanopb
    pb_istream_t istream = pb_istream_from_buffer(buffer, total_bytes_encoded);
    if (!pb_decode(&istream, SimpleMessage_fields, &msg))
    {
        const char * error = PB_GET_ERROR(&istream);
        printf("pb_decode error: %s", error);
        return;
    }

    printf("Bytes decoded: %d", total_bytes_encoded - istream.bytes_left);
}

If you have a repeated struct inside your message, your callback will not use nanopb primitive functions (like pb_decode_varint32 above), but again pb_decode for each concrete message type. Your callback can also attach new callbacks to those nested structs, if needed.

Answer 2

To complement Groo's answer, here are answers to your specific questions.

1. Now, where and what should the function contain?

Groo provided good explanation of the callback functions. The network_server example in nanopb repository also uses callbacks and can be a useful reference: network_server/server.c network_server/client.c

2. Where does the data itself being stored?

Wherever you want! The whole point of nanopb's callbacks is that it gives you full flexibility in deciding how to store your data. In some cases you may want to even process the data on the fly, not storing it anywhere.

For example, the network_server example above gets the filenames from filesystem and sends them to the network directly - this way it can handle any amount of files without requiring much memory.

3. How can I access it/ assign new data to it?

Now this is the downside of callbacks - you'll have to implement your own access and allocation functions for whatever storage you use. That's why for the most common cases, either static allocation (with fixed maximum size) or dynamic allocation (which malloc()s required amount of memory) are more convenient.