Is "google/protobuf/struct.proto" the best way to send dynamic JSON over GRPC?

Question

I have a written a simple GRPC server and a client to call the server (both in Go). Please tell me if using golang/protobuf/struct is the best way to send a dynamic JSON with GRPC. In the example below, earlier I was creating Details as a map[string]interface{} and serializing it. Then I was sending it in protoMessage as bytes and was de-serializing the message on the server side.

Is it the best/efficient way to do it or should I define Details as a struct in my proto file?

Below is User.proto file

syntax = "proto3";
package messages;
import "google/protobuf/struct.proto";

service UserService {
    rpc SendJson (SendJsonRequest) returns (SendJsonResponse) {}
}

message SendJsonRequest {
    string UserID = 1;
    google.protobuf.Struct Details = 2;
}

message SendJsonResponse {
    string Response = 1;
}

Below is client.go file

package main
import (
    "context"
    "flag"
    pb "grpc-test/messages/pb"
    "log"
    "google.golang.org/grpc"
)

func main() {
    var serverAddr = flag.String("server_addr", "localhost:5001", "The server address in the format of host:port")
    opts := []grpc.DialOption{grpc.WithInsecure()}
    conn, err := grpc.Dial(*serverAddr, opts...)
    if err != nil {
        log.Fatalf("did not connect: %s", err)
    }
    defer conn.Close()

    userClient := pb.NewUserServiceClient(conn)
    ctx := context.Background()

    sendJson(userClient, ctx)
}

func sendJson(userClient pb.UserServiceClient, ctx context.Context) {
    var item = &structpb.Struct{
        Fields: map[string]*structpb.Value{
            "name": &structpb.Value{
                Kind: &structpb.Value_StringValue{
                    StringValue: "Anuj",
                },
            },
            "age": &structpb.Value{
                Kind: &structpb.Value_StringValue{
                    StringValue: "Anuj",
                },
            },
        },
    }

    userGetRequest := &pb.SendJsonRequest{
        UserID: "A123",
        Details: item,
    }

    res, err := userClient.SendJson(ctx, userGetRequest)
}

Answer 1

Based on this proto file.

syntax = "proto3"; package messages; import "google/protobuf/struct.proto";  service UserService {     rpc SendJson (SendJsonRequest) returns (SendJsonResponse) {} }  message SendJsonRequest {     string UserID = 1;     google.protobuf.Struct Details = 2; }  message SendJsonResponse {     string Response = 1; } 

I think it is a good solution to use the google.protobuf.Struct type.

The folks with their answers, helped me a lot at the beginning, so I would like to say thanks for your work! :) I really appreciate both solutions! :) On the other hand, I think I found a better one to produce these kinds of Structs.

Anuj's Solution

This is a little bit overcomplicated but it can work.

var item = &structpb.Struct{     Fields: map[string]*structpb.Value{         "name": &structpb.Value{             Kind: &structpb.Value_StringValue{                 StringValue: "Anuj",             },         },         "age": &structpb.Value{             Kind: &structpb.Value_StringValue{                 StringValue: "Anuj",             },         },     }, } 

Luke's Solution

This is a shorter one but still require more conversion than necessary. map[string]interface{} -> bytes -> Struct

m := map[string]interface{}{   "foo":"bar",   "baz":123, } b, err := json.Marshal(m) s := &structpb.Struct{} err = protojson.Unmarshal(b, s) 

The solution from my perspective

My solution will use the official functions from the structpb package which is pretty well documented and user friendly.

Documentation: https://pkg.go.dev/google.golang.org/protobuf/types/known/structpb

For example, this code creates a *structpb.Struct via the function that was designed to do this.

m := map[string]interface{}{     "name": "Anuj",     "age":  23, }  details, err := structpb.NewStruct(m) // Check to rules below to avoid errors if err != nil {     panic(err) }  userGetRequest := &pb.SendJsonRequest{     UserID: "A123",     Details: details, } 

One of the most important thing, that we should keep in mind when we are building Struct from a map[string]interface{} is this:

https://pkg.go.dev/google.golang.org/protobuf/types/known/structpb#NewValue

// NewValue constructs a Value from a general-purpose Go interface. // //  ╔════════════════════════╤════════════════════════════════════════════╗ //  ║ Go type                │ Conversion                                 ║ //  ╠════════════════════════╪════════════════════════════════════════════╣ //  ║ nil                    │ stored as NullValue                        ║ //  ║ bool                   │ stored as BoolValue                        ║ //  ║ int, int32, int64      │ stored as NumberValue                      ║ //  ║ uint, uint32, uint64   │ stored as NumberValue                      ║ //  ║ float32, float64       │ stored as NumberValue                      ║ //  ║ string                 │ stored as StringValue; must be valid UTF-8 ║ //  ║ []byte                 │ stored as StringValue; base64-encoded      ║ //  ║ map[string]interface{} │ stored as StructValue                      ║ //  ║ []interface{}          │ stored as ListValue                        ║ //  ╚════════════════════════╧════════════════════════════════════════════╝ // // When converting an int64 or uint64 to a NumberValue, numeric precision loss // is possible since they are stored as a float64. 

For example, if you would like to produce a Struct that has a string list in its JSON form, you should create the following map[string]interface{}

m := map[string]interface{}{     "name": "Anuj",     "age":  23,     "cars": []interface{}{         "Toyota",         "Honda",         "Dodge",     } } 

Sorry for the long post, I hope it makes your work easier with proto3 and Go! :)

Answer 2

I ended up using a two-step conversion with protojson, from map to json to struct:

m := map[string]interface{}{
  "foo":"bar",
  "baz":123,
}
b, err := json.Marshal(m)
s := &structpb.Struct{}
err = protojson.Unmarshal(b, s)

I don't find it elegant but could not really find any official documentation of how to do this differently. I also prefer to produce structures using "official" functions rather than trying to build a structure myself.