Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

protobuf.js translate proto file to JSON descriptor, repeated gets lost

I'm using Protobuf.js to build a node package, containing our protocol and offering encode and decode functionality to Proto Messages defined in this package. I would be fine with using .proto files (The loading of .proto files happens at runtime) , but since the module needs to be usable on the client side and I can't pack the .proto files to my resolved .js-file (built with browserify), I need to use a way, that enables the packaging in the build.js.

Enter JSON Descriptors.

var jsonDescriptor = require("./awesome.json"); // exemplary for node

var root = protobuf.Root.fromJSON(jsonDescriptor);

The json file can be packed up (requirement resolved by browserify). Proto Type Defintions also are possible in .json

I translated my .proto file into a .json file and tried it with my example data. Unfortunately it failed with the repeated fields.

The .proto file looks kind of like this:

message Structure {
    map <int32, InnerArray> blocks = 1;
}

message Inner{
    int32 a = 1;
    int32 b = 2;
    bool c = 3;
}

message InnerArray{
    repeated Inner inners = 1;
}   

Which I translated into this JSON Descriptor

{
  "nested": {
    "Structure": {
      "fields": {
        "blocks": {
          "type": "InnerArray",
          "id": 1,
          "map" : true,
          "keyType" : "int32"
        }
      }
    },
    "InnerArray" : {
        "fields": {
            "inners" : {
                "repeated" : true,
                "type" : "Inner",
                "id" : 1
            }
        }
    },
    "Inner" : {
        "fields" : {
            "a" : {
                "type" : "int32",
                "id" : 1
            },
            "b" : {
                "type" : "int32",
                "id" : 2
            },
            "c" : {
                "type" : "bool",
                "id" : 3
            }
        }
    }
  }
}

If I'm not mistaken there is the required attribute for a field.

When I encode and decode my example data it stops at the repeated field: (note that the map works fine).

{
  "blocks": {
    "0": {
      "inners": {}
    },
    ...

I also examined my root to find out how the loaded type looks and it looks exactly like my defintion EXCEPT that repeated is missing:

"InnerArray" : {
            "fields": {
                "inners" : {
                    "type" : "Inner",
                    "id" : 1
                }
            }
        },

How do I define a repeated field correctly in a JSON Descriptor?

If there is a way to pre include proto files and not load them at runtime, so that i can wrap them up with browserify, I would accept this as a solution, too.

like image 511
Bernd Strehl Avatar asked Jun 02 '17 12:06

Bernd Strehl


People also ask

What does Repeated mean in Protobuf?

repeated : this field can be repeated any number of times (including zero) in a well-formed message. The order of the repeated values will be preserved.

Is Protobuf faster than JSON?

TL;DR — encoding and decoding string-intensive data in JavaScript is faster with JSON than it is with protobuf. When you have structured data in JavaScript, which needs to be sent over the network (for another microservice for example) or saved into a storage system, it first needs to be serialized.

Does JavaScript support Protobuf?

Protocol buffers, or Protobuf, is a binary format created by Google to serialize data between different services. Google made this protocol open source and now it provides support, out of the box, to the most common languages, like JavaScript, Java, C#, Ruby and others.


1 Answers

After browsing through the code, I found that you can not set required in a JSON Descriptor. The correct way is to set "rule": "repeated"; since a field is set with a Field Descriptor

like image 159
Bernd Strehl Avatar answered Sep 28 '22 06:09

Bernd Strehl