Failed to obtain the location of the Google Cloud Storage bucket

I am trying to transfer data from S3 to GCS by using a Java client but I got this error.

Failed to obtain the location of the Google Cloud Storage (GCS) bucket ___ due to insufficient permissions. Please verify that the necessary permissions have been granted.

I am using a service account with the Project Owner role, which should grant unlimited access to all project resources.

Where is my Google Cloud bucket?

In the Google Cloud console, go to the Cloud Storage Buckets page. In the bucket list, find the bucket you want to verify, and check its column headers for the metadata you want to view.

How do I change the location of my Google Cloud bucket?

You cannot change a bucket's location after it's created, but you can move your data to a bucket in a different location.

Google Transfer Service is using an internal service account to move the data back and forth. This account is created automatically and should not be confused with the service accounts you create yourself.

You need to give this user a permission called "Legacy bucket writer".

This is written in the documentation, but it's VERY easy to miss:

https://cloud.google.com/storage-transfer/docs/configure-access