google.cloud.storage allows access but gcsfs does not

Question

I would like to connect to a bucket on Google Cloud storage using gcsfs. So far I have only been using the native google.cloud module but it turns out I actually need file like objects for a certain application so had to switch over.

My GOOGLE_APPLICATION_CREDENTIALS environment variable points to a JSON file on my local filesystem. Using google.cloud, I am able to access a bucket in cloud storage with no issues. However using gcsfs I am not. Here's some code to show what I mean:

from google.cloud import storage
import gcsfs
import google.auth

storage_client = storage.Client()
bucket = storage_client.bucket(...)
blob = bucket.blob(f'data/audio_wav/test.wav')
bts = blob.download_as_bytes()

credentials, _ = google.auth.default()
fs = gcsfs.GCSFileSystem(project=..., token=credentials)
folders = fs.ls('data') #throws error

As commented, the very last line throws what appears to be an authentication error:

google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', {'error': 'invalid_scope', 'error_description': 'Invalid OAuth scope or ID token audience provided.'})

I would think that if google.cloud.storage accepts my credentials, then so should gcsfs. Why do I get the above error, and how can I fix it?

Alternatively, is there a way to open a file like object using google.cloud.storage and not gcsfs?

JSON file

{
  "type": "service_account",
  "project_id": ...,
  "private_key_id": ....,
  "private_key": ...,
  "client_email": "[email protected]",
  "client_id": "113487991001810308893",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ai-dj-884%40ai-dj-76527.iam.gserviceaccount.com"
}

Answer 1

As @John Hanley suggested in comments modify the mode of authentication (i.e., “token”) from "token=credentials" to "token=None", to use your default gcloud credentials or, attempt to get credentials from the Google metadata service, or fall back to anonymous access.

Refer to Credentials for more information.

To open a file using ‘“google.cloud.storage’” instead of "gcsfs" you can use the below method,

#create storage client  
storage_client = storage.Client('json-file-path’')  
#get bucket with name  
bucket = storage_client.get_bucket('bucket-name')  
#get bucket data as blob  
blob = bucket.get_blob('file-name’)