Google Cloud Storage giving ServiceUnavailable: 503 exception Backend Error

Question

I'm trying to upload a file to Google Cloud Storage Bucket. While making it public, intermittently I'm getting this exception from Google. This error comes almost once in 20 uploads. google.api_core.exceptions.ServiceUnavailable: 503 GET https://www.googleapis.com/storage/v1/b/bucket_name/o/folder_name%2FPolicy-APP-000456384.2019-05-16-023805.pdf/acl: Backend Error

I'm using python3 and have tried updating the version of google-cloud-storage to 1.15.0 but it didn't help.

class GoogleStorageHelper:
    def __init__(self, project_name):
        self.client = storage.Client(project=project_name)

    def upload_file(self, bucket_name, file, file_name, content_type, blob_name, is_stream):
        safe_file_name = self.get_safe_filename(file_name)
        bucket = self.client.bucket(bucket_name)
        blob = bucket.blob(safe_file_name)
        if is_stream:
            blob.upload_from_string(file, content_type=content_type)
        else:
            blob.upload_from_filename(file, content_type=content_type)
        blob.make_public() // Getting Error here
        url = blob.public_url
        if isinstance(url, six.binary_type):
            url = url.decode('utf-8')
        logger.info('File uploaded, URL: {}'.format(url))
        return url

    @staticmethod
    def get_safe_filename(file_name):
        basename, extension = file_name.rsplit('.', 1)
        return '{0}.{1}.{2}'.format(basename, datetime.now().strftime('%Y-%m-%d-%H%M%S'), extension)

Have you faced this kind of problem and solved it? Or have any ideas to fix this issue?

Answer 1

This is a known issue recently with GCS using Python make_public() method. The problem is now being worked on by the GCS team.

I'd suggest, as a quick mitigation strategy, to enable retries. This documentation could be helpful in setting up Retry Handling Strategy.

Answer 2

This one is a bit tricky. I ran into the same issue and found the Python API client doesn't enable retries for the upload_from_string() method.

All upload_from_string() does is call the upload_from_file() method, which has retries but the implementation ignores retries.

def upload_from_string(self, 
                       data, 
                       content_type="text/plain", 
                       client=None, 
                       predefined_acl=None):
    data = _to_bytes(data, encoding="utf-8")
    string_buffer = BytesIO(data)
    self.upload_from_file(
        file_obj=string_buffer,
        size=len(data),
        content_type=content_type,
        client=client,
        predefined_acl=predefined_acl,
    )    

You can hack the upload_from_string() method by using the upload_from_file() implementation, adding retries:

from google.cloud._helpers import _to_bytes
from io import BytesIO
from google.cloud.storage import Blob

def upload_from_string(
    data, file_path, bucket, client, content_type, num_retries
):
    data = _to_bytes(data, encoding="utf-8")
    string_buffer = BytesIO(data)    

    blob = Blob(file_path, bucket)

    blob.upload_from_file(
        file_obj=string_buffer,
        size=len(data),
        client=client,
        num_retries=num_retries,
        content_type=content_type
    )