Is there a limit to the length of metadata values in Google Cloud Storage?

Question

When uploading a file to Google Cloud Storage, there is a custom data field metadata.

Google's example is fairly short:

var metadata = {
  contentType: 'application/x-font-ttf',
  metadata: {
    my: 'custom',
    properties: 'go here'
  }
};

file.setMetadata(metadata, function(err, apiResponse) {});

Is there a maximum for how big GCS will allow for the metadata object, should I wish to store manifests of tar and zip files, or a few hundred KB in there?

Answer 1

Using the following command to upload set metadata in GCS:

$ echo '{"metadata": {"large": "' > body ; tr -dC '[:print:]' < /dev/urandom | tr -d '\\"' | head -c SIZE_OF_METADATA_IN_BYTES >> body ;  echo '"}}' >> body; curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -X PATCH -H "Content-type: application/json" -d @body -o return_body https://www.googleapis.com/storage/v1/b/manifest-geode-357/o/empty

I Find that above 2097KB header the service returns "HTTP 413 Request Too Large" and metadata is not set. Below that level it is set as expected. If I use more compressible input (e.g. the output of yes), I can get more data, but the cut-off is at the same content-length value (which is post-compression). As 2097KB == 2MiB almost exactly, I expect the true limitation is the entire HTTP request must fit in 2MiB.

---

However Brandon's comment is correct: this is not a great idea for a whole catalog of reasons:

1. This will cause you to consume more bandwidth (with associated performance and cost penalty)
2. You won't save any money on storage costs (as metadata is still charged for).
3. It relies on undocumented behaviour which might be changed by Google without any notice.
4. Unlike real object data, there is no resumable-behaviour on upload, so errors hit you worse.
5. There is no checksum to verify integrity during the upload process.
6. It's likely that many client libraries store metadata in-memory rather than on disk or keep multiple copies, so you are more likely to see memory pressure in your application.

Simply storing the manifest in a separate object solves all of these issues. You could store thie location of the manifest in the metadata, and get the benefit of both options.