Why required and optional is removed in Protocol Buffers 3

Question

The usefulness of required has been at the heart of many a debate and flame war. Large camps have existed on both sides. One camp liked guaranteeing a value was present and was willing to live with its limitations but the other camp felt required dangerous or unhelpful as it can't be safely added nor removed.

Let me explain more of the reasoning why required fields should be used sparingly. If you are already using a proto, you can't add a required field because old application's won't be providing that field and applications in general don't handle the failure well. You can make sure that all old applications are upgraded first, but it can be easy to make a mistake and it doesn't help if you are storing the protos in any datastore (even short-lived, like memcached). The same sort of situation applies when removing a required field.

Many required fields were "obviously" required until... they weren't. Let's say you have an id field for a Get method. That is obviously required. Except, later you might need to change the id from int to string, or int32 to int64. That requires adding a new muchBetterId field, and now you are left with the old id field that must be specified, but eventually is completely ignored.

When those two problems are combined, the number of beneficial required fields becomes limited and the camps argue over whether it still has value. The opponents of required weren't necessarily against the idea, but its current form. Some suggested developing a more expressive validation library that could check required along with something more advanced like name.length > 10, while also making sure to have a better failure model.

Proto3 overall seems to favor simplicity, and required removal is simpler. But maybe more convincing, removing required made sense for proto3 when combined with other features, like removal of field presence for primitives and removal of overriding default values.

I'm not a protobuf developer and am in no way authoritative on the subject, but I still hope that the explanation is useful.

You can find the explanation in this protobuf Github issue:

We dropped required fields in proto3 because required fields are generally considered harmful and violating protobuf's compatibility semantics. The whole idea of using protobuf is that it allows you to add/remove fields from your protocol definition while still being fully forward/backward compatible with newer/older binaries. Required fields break this though. You can never safely add a required field to a .proto definition, nor can you safely remove an existing required field because both of these actions break wire compatibility. For example, if you add a required field to a .proto definition, binaries built with the new definition won't be able to parse data serialized using the old definition because the required field is not present in old data. In a complex system where .proto definitions are shared widely across many different components of the system, adding/removing required fields could easily bring down multiple parts of the system. We have seen production issues caused by this multiple times and it's pretty much banned everywhere inside Google for anyone to add/remove required fields. For this reason we completely removed required fields in proto3.

After the removal of "required", "optional" is just redundant so we removed "optional" as well.

Optional fields were returned in protobuf 3.15