What would be the best approach to converting protoc generated structs from bson structs?

Question

I'm writing a RESTful API in Golang, which also has a gRPC api. The API connects to a MongoDB database, and uses structs to map out entities. I also have a .proto definition which matches like for like the struct I'm using for MongoDB.

I just wondered if there was a way to share, or re-use the .proto defined code for the MongoDB calls also. I've noticed the strucs protoc generates has json tags for each field, but obviously there aren't bson tags etc.

I have something like...

// Menu -
type Menu struct {
    ID          bson.ObjectId      `json:"id" bson"_id"`
    Name        string             `json:"name" bson:"name"`
    Description string             `json:"description" bson:"description"`
    Mixers      []mixers.Mixer     `json:"mixers" bson:"mixers"`
    Sections    []sections.Section `json:"sections" bson:"sections"`
}

But then I also have protoc generated code...

type Menu struct {
    Id          string     `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
    Name        string     `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
    Description string     `protobuf:"bytes,3,opt,name=description" json:"description,omitempty"`
    Mixers      []*Mixer   `protobuf:"bytes,4,rep,name=mixers" json:"mixers,omitempty"`
    Sections    []*Section `protobuf:"bytes,5,rep,name=sections" json:"sections,omitempty"`
}

Currently I'm having to convert between the two structs depending what I'm doing. Which is tedious and I'm probably quite a considerable performance hit. So is there a better way of converting between the two, or re-using one of them for both tasks?

Answer 1

Having lived with this same issue, there's a couple methods of solving it. They fall into two general methods:

1. Use the same data type
2. Use two different struct types and map between them

If you want to use the same data type, you'll have to modify the code generation

You can use something like gogoprotobuf which has an extension to add tags. This should give you bson tags in your structs.

You could also post-process your generated files, either with regular expressions or something more complicated involving the go abstract syntax tree.

If you choose to map between them:

1. Use reflection. You can write a package that will take two structs and try to take the values from one and apply it to another. You'll have to deal with edge cases (slight naming differences, which types are equivalent, etc), but you'll have better control over edge cases if they ever come up.

2. Use JSON as an intermediary. As long as the generated json tags match, this will be a quick coding exercise and the performance hit of serializing and deserializing might be acceptable if this isn't in a tight loop in your code.

3. Hand-write or codegen mapping functions. Depending on how many structs you have, you could write out a bunch of functions that translate between the two.

At my workplace, we ended up doing a bit of all of them: forking the protoc generator to do some custom tags, a reflection based structs overlay package for mapping between arbitrary structs, and some hand-written ones in more performance sensitive or less automatable mappings.

Answer 2

I have played with it and have a working example with:

github.com/gogo/protobuf v1.3.1
go.mongodb.org/mongo-driver v1.4.0
google.golang.org/grpc v1.31.0

First of all I would like to share my proto/contract/example.proto file:

syntax = "proto2";

package protobson;

import "gogoproto/gogo.proto";

option (gogoproto.sizer_all) = true;
option (gogoproto.marshaler_all) = true;
option (gogoproto.unmarshaler_all) =  true;
option go_package = "gitlab.com/8bitlife/proto/go/protobson";

service Service {
    rpc SayHi(Hi) returns (Hi) {}
}

message Hi {
    required bytes id = 1 [(gogoproto.customtype) = "gitlab.com/8bitlife/protobson/custom.BSONObjectID", (gogoproto.nullable) = false, (gogoproto.moretags) = "bson:\"_id\""] ;
    required int64 limit = 2  [(gogoproto.nullable) = false, (gogoproto.moretags) = "bson:\"limit\""] ;
}

It contains a simple gRPC service Service that has SayHi method with request type Hi. It includes a set of options: gogoproto.sizer_all, gogoproto.marshaler_all, gogoproto.unmarshaler_all. Their meaning you can find at extensions page. The Hi itself contains two fields:

1. id that has additional options specified: gogoproto.customtype and gogoproto.moretags
2. limit with only gogoproto.moretags option

BSONObjectID used in gogoproto.customtype for id field is a custom type that I defined as custom/objectid.go:

package custom

import (
    "go.mongodb.org/mongo-driver/bson/bsontype"
    "go.mongodb.org/mongo-driver/bson/primitive"
)

type BSONObjectID primitive.ObjectID

func (u BSONObjectID) Marshal() ([]byte, error) {
    return u[:], nil
}

func (u BSONObjectID) MarshalTo(data []byte) (int, error) {
    return copy(data, (u)[:]), nil
}

func (u *BSONObjectID) Unmarshal(d []byte) error {
    copy((*u)[:], d)
    return nil
}

func (u *BSONObjectID) Size() int {
    return len(*u)
}

func (u *BSONObjectID) UnmarshalBSONValue(t bsontype.Type, d []byte) error {
    copy(u[:], d)
    return nil
}

func (u BSONObjectID) MarshalBSONValue() (bsontype.Type, []byte, error) {
    return bsontype.ObjectID, u[:], nil
}

It is needed because we need to define a custom marshaling and un-marshaling methods for both: protocol buffers and mongodb driver. This allows us to use this type as an object identifier in mongodb. And to "explaine" it to mongodb driver I marked it with a bson tag by using (gogoproto.moretags) = "bson:\"_id\"" option in proto file.

To generate source code from the proto file I used:

protoc \
    --plugin=/Users/pstrokov/go/bin/protoc-gen-gogo \
    --plugin=/Users/pstrokov/go/bin/protoc-gen-go \
    -I=/Users/pstrokov/Workspace/protobson/proto/contract \
    -I=/Users/pstrokov/go/pkg/mod/github.com/gogo/[email protected] \
    --gogo_out=plugins=grpc:. \
    example.proto

I have tested it on my MacOS with running MongoDB instance: docker run --name mongo -d -p 27017:27017 mongo:

package main

import (
    "context"
    "log"
    "net"
    "time"

    "gitlab.com/8bitlife/protobson/gitlab.com/8bitlife/proto/go/protobson"
    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
    "google.golang.org/grpc"
)

type hiServer struct {
    mgoClient *mongo.Client
}

func (s *hiServer) SayHi(ctx context.Context, hi *protobson.Hi) (*protobson.Hi, error) {
    collection := s.mgoClient.Database("local").Collection("bonjourno")
    res, err := collection.InsertOne(ctx, bson.M{"limit": hi.Limit})
    if err != nil { panic(err) }
    log.Println("generated _id", res.InsertedID)

    out := &protobson.Hi{}
    if err := collection.FindOne(ctx, bson.M{"_id": res.InsertedID}).Decode(out); err != nil { return nil, err }
    log.Println("found", out.String())
    return out, nil
}

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    lis, err := net.Listen("tcp", "localhost:0")
    if err != nil { log.Fatalf("failed to listen: %v", err) }
    clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
    clientOptions.SetServerSelectionTimeout(time.Second)
    client, err := mongo.Connect(ctx, clientOptions)
    if err != nil { log.Fatal(err) }
    if err := client.Ping(ctx, nil); err != nil { log.Fatal(err) }
    grpcServer := grpc.NewServer()
    protobson.RegisterServiceServer(grpcServer, &hiServer{mgoClient: client})
    go grpcServer.Serve(lis); defer grpcServer.Stop()
    conn, err := grpc.Dial(lis.Addr().String(), grpc.WithInsecure())
    if err != nil { log.Fatal(err) }; defer conn.Close()
    hiClient := protobson.NewServiceClient(conn)
    response, err := hiClient.SayHi(ctx, &protobson.Hi{Limit: 99})
    if err != nil { log.Fatal(err) }
    if response.Limit != 99 { log.Fatal("unexpected limit", response.Limit) }
    if response.Id.Size() == 0 { log.Fatal("expected a valid ID of the new entity") }
    log.Println(response.String())
}

Sorry for the formatting of the last code snippet :) I hope this can help.