Backward and forward compatibility issues with protobufs in Google Pub/Sub

Question

We use protocol buffers both for gRPC server-to-server communication and for publishing messages to Pub/Sub.

Pub/Sub is fairly sensitive to schema changes, not allowing any schema changes that would result in compatibility problems.

But I recently saw a change that was considered as creating compatibility problems that I would not have expected.

I changed an enum in a message from something like

message Refund {
  optional string refund_id = 1;
  .
  .
  .
  optional RefundStatus refund_status = 12;
  enum RefundStatus {
    REFUND_STATUS_UNSPECIFIED = 0;
    REFUND_STATUS_NOT_YET_DEFINED = 1;
    REFUND_STATUS_REFUND_INITIATED = 2;
    REFUND_STATUS_REFUND_FAILED = 3;
    REFUND_STATUS_REFUNDED = 4;
  }
}

to something like

message Refund {
  optional string refund_id = 1;
  .
  .
  .
  optional RefundStatus refund_status = 12;
  enum RefundStatus {
    REFUND_STATUS_UNSPECIFIED = 0;
    REFUND_STATUS_NOT_YET_DEFINED = 1;
    REFUND_STATUS_REFUND_INITIATED = 2;
    REFUND_STATUS_REFUND_FAILED = 3;
    REFUND_STATUS_REFUNDED = 4;
    REFUND_STATUS_REFUND_CREATED = 5;
  }
}

That is, I simply added a new enum level, called REFUND_CREATED. To my understanding of protobufs, this is completely backward and forward compatible: enums are represented as integers, and I can provide the integer 5 to a schema's enum that doesn't have the level 5 (yet). Information is lost, but nothing breaks.

Yet, I got the following error when trying to update the Pub/Sub schema (using Terraform):

│ Error: Error updating Schema "projects/<my_project>/schemas/refund": googleapi: Error 400: Compatibility checking failed to commit a schema revision for (schema="projects/<my_project>/schemas/refund"). (reason="Revision is incompatible with previous revision: <value>. Failed with error: refund.v0.Refund.REFUND_STATUS_REFUND_CREATED = 5 is missing in refund.v0.Refund.RefundStatus). Refer to https://cloud.google.com/pubsub/docs/schemas for more information.

Why is this happening? Is it actually a breaking change to add enum levels to a Pub/Sub schema that is using a protobuf to define its schema? We will be updating protobuf levels all the time, which means we'll be making breaking changes all the time. This makes the Pub/Sub schema almost unusable. Or, it makes it very difficult to use enums in the schema.

How should this best be handled? Should I just move all enums back to strings until our schemas are very stable?

Answer 1

This is a known issue and is being publicly tracked here: https://issuetracker.google.com/issues/373461924.

I've added a comment to the bug and I'll discuss it with the rest of the team, but we cannot make any promises at this point in time. Sorry for the inconvenience.