Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can I retire a required protobuf field?

I have a client-server application where the server transmits serialized objects in protobuf format to a client, and I would like to retire a required field. Unfortunately I can't change both client and server at the same time to use the new .proto definition.

If I change a required field to be optional, but only for code that serializes messages (i.e. deserializing code has not been rebuilt and still thinks it's a required field), can I continue to publish messages that can be deserialized as long as I populate a value for the now-optional field?

(It appears to be fine to do so, at least for a few trivial cases I experimented with (only using Java), but I'm interested if it's a generally sensible approach, and whether there are any edge cases etc I should worry about).

Motivation: My goal is to retire a required field in a client-server application where the server publishes messages that are deserialized by the client. My intended approach is:

  1. Change required field to optional on the trunk.
  2. If it's necessary to deploy new server code (for unrelated features/fixes), ensure that the optional field continues to be populated in the message.
  3. Gradually deploy updated code for all clients (this will take time as it requires involvement of other teams with their own release schedules)
  4. Confirm that all clients have been updated.
  5. Begin to retire (i.e. not populate) the optional field.
like image 788
bacar Avatar asked Dec 03 '12 00:12

bacar


People also ask

Can you remove field from Protobuf?

Removing fields is fine, although you might want to mark it reserved so that nobody reuses it in an incompatible way. New code with old data (with the field) will silently ignore it; old code with new data will just load without the field populated, since everything in proto3 is implicitly optional .

Why did proto3 remove optional?

We have seen production issues caused by this multiple times and it's pretty much banned everywhere inside Google for anyone to add/remove required fields. For this reason we completely removed required fields in proto3. After the removal of "required", "optional" is just redundant so we removed "optional" as well.

Can Protobuf fields be null?

Turns out, protobuf does not allow setting null values on any object field.

Can I rename Protobuf fields?

Renaming a field - With Protobuf content, the field names are only used in generated code. The field number is used to identify fields on the network. Renaming a field isn't a protocol breaking change for Protobuf. However, if a server is using JSON content then renaming a field is a breaking change.


2 Answers

According to the encoding format documentation, whether a field is required or not is not encoded in serialized byte stream itself. That is, optional or required makes no difference in the encoded serialized message.

I've confirmed this in practice, using the Java generated code, by writing serialized messages to disk and comparing the output - using a message containing all of the supported primitive types as well as fields representing other types.

like image 157
bacar Avatar answered Sep 28 '22 11:09

bacar


As long as the field is set, using the parseFrom(byte[]) method to deserialize will still work, because the byte[] will be the same.

However, one would wonder why you would change the field from required to optional until you are ready to allow it to be optional? Basically you are just making it "optional" in the .proto file, but you are enforcing that it is required by always populating it. Just a thought.

like image 43
CaTalyst.X Avatar answered Sep 28 '22 09:09

CaTalyst.X