How to use protobuf reflection to guarantee deterministic serialisation

Question

Proto3 release notes states:

The deterministic serialization is, however, NOT canonical across languages; it is also unstable across different builds with schema changes due to unknown fields. Users who need canonical serialization, e.g. persistent storage in a canonical form, fingerprinting, etc, should define their own canonicalization specification and implement the serializer using reflection APIs rather than relying on this API.

What I would like to achieve is to have a deterministic serialisation of protobuf message to carry a crypto signature along with it. As I understand due to differences in serialisers binary data could differ and signature would become invalid.

package Something

message Request {
  Payload payload = 1;

  // signature of serialised payload
  bytes signature = 2;

  message Payload {
    string user_id_from = 1;
    uint64 amount = 2;
    string user_id_to = 3;
  }
}

What is the way to do this using protobuf reflection?

Answer 1

This doesn't answer the question directly, but may solve your issue: don't store the payload as a message, but store the serialized bytes alongside with the signature.

message Request {
  // Serialized Payload message.
  bytes payload = 1;

  // signature of serialised payload
  bytes signature = 2;
}

message Payload {
  string user_id_from = 1;
  uint64 amount = 2;
  string user_id_to = 3;
}

This may be a little less convenient to work with in code, but has the advantage of preserving all the forwards and backwards-compatibility guarantees of protobuf.

It also frees you from serializing the message twice when writing it (once as a subfield, once to get the signature).