AWS buckets and regions

Question

The application is using carrierwave in conjunction with the carrierwave-aws gem. It hit snags when migrating rails versions (bumped up to 4.2), ruby versions (2.2.3) and redeployed to the same staging server.

The AWS bucket was initially created in the free tier, thus Oregon, us-west-2. However, I have found that all the S3 files have the property which links to eu-west-1. Admittedly I've been tinkering around and considered using the eu-west-1 region. However I do not recall making any config changes - not even sure it is allowed in the free tier...

So yes, I've had to configure my uploads initializer with:

config.asset_host = 'https://s3-eu-west-1.amazonaws.com/theapp'
config.aws_credentials = {
  region:            'eu-west-1'
}

Now the console for AWS is accessible with a URL that includes region=us-west-2

I do not understand how this got to be and am looking for suggestions.

Answer 1

In spite of appearances, and AWS account doesn't have a "home" (native) region.

The console defaults to us-west-2 (Oregon), and conventional wisdom suggests that this is a region where AWS has the most available/spare resources, lower operational costs, lower pricing for customers, and fewest constraints for growth, so that in the event a user does not have enough information at hand to actively select a region where they deploy services, Oregon will be used by default.

But for each account, no particular region has any special standing. If you switch regions in the console, the console will tend to open up to the same region next time.

Most AWS services -- EC2, SQS, SNS, RDS (to name a few) are strictly regional: the regions are independent and not connected together¹, in the interest of reliability and survivability. When you're in a given region in the console, you can only see EC2 resources in that region, SQS queues in that region, SNS topics in that region, etc. To see your resources in other regions, you switch regions in the console.

When making API requests to these services, you're using an endpoint in the region and your credentials also include the region.

Other services are global, with centralized administration -- examples here are CloudFront, IAM, and Route 53 hosted zones. When you make requests to these services, you always use the region "us-east-1" because that's the home location of those services' central, global administration. These tend to be services were a partitioning event (where one part of the global network is isolated from another). Administrative changes are replicated out around the world, but after the provisioning is replicated, the regional installations can operate autonomously without major service impacts. When you select these services in the console, you'll note that the region changes to "Global."

S3 is a hybrid that is different from essentially all of the others. When you select S3 in the console, you'll notice that the console region also changes to show "Global" and you can see all of your buckets, like other global services. S3 has independently operating regions, but a global namespace. The regions are logically connected and can communicate administrative messages among themselves and can transfer data across regions (but only when you do this deliberately -- otherwise, data remains in the region where you stored it).

Unlike the other global services, S3 does not have a single global endpoint that can handle every possible request.

Each time you create a bucket, you choose the region where you want to bucket to live. Subsequent requests related to that bucket have to be submitted to the bucket's region, and must have authorization credentials for the correct region.

If you submit a request to another S3 region's endpoint for that bucket, you'll receive an error telling you the correct region for the bucket.

< HTTP/1.1 301 Moved Permanently
< x-amz-bucket-region: us-east-1
< Server: AmazonS3
<Error>
  <Code>PermanentRedirect</Code>
  <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
  <Bucket>...</Bucket>
  <Endpoint>s3.amazonaws.com</Endpoint>
  <RequestId>...</RequestId>
  <HostId>...</HostId>
</Error>

Conversely, if you send an S3 request to the correct endpoint but using the wrong region in your authentication credentials, you'll receive a different error for similar reasons:

< HTTP/1.1 400 Bad Request
< x-amz-bucket-region: us-west-2
< Server: AmazonS3
<
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AuthorizationQueryParametersError</Code>
  <Message>Error parsing the X-Amz-Credential parameter; the region 'eu-west-1' is wrong; expecting 'us-west-2'</Message>
  <Region>us-west-2</Region>
  <RequestId>...</RequestId>
  <HostId>...</HostId>
</Error>

Again, this region is the region where you created the bucket, or the default "US Standard" (us-east-1). Once a bucket has been created, it can't be moved to a different region. The only way to "move" a bucket to a different region without the name changing is to remove all the files from the bucket, delete the bucket (you can't delete a non-empty bucket), wait a few minutes, and create the bucket in the new region. During the few minutes that S3 requires before the name is globally available after deleting a bucket, it's always possible that somebody else could take the bucket name for themselves... so choose your bucket region carefully.

_{S3 API interactions were edited and reformatted for clarity; some unrelated headers and other content was removed.}

---

¹ not connected together seems (at first glance) to be contradicted in the sense that -- for example -- you can subscribe an SQS queue in one region to an SNS topic in another, you can replicate RDS from one region to another, and you can transfer EBS snapshots and AMIs from one region to another... but these back-channels are not under discussion, here. The control planes of the services in each region are isolated and independent. A problem with the RDS infrastructure in one region might disrupt replication to RDS in another region, but would not impact RDS operations in the other region. An SNS outage in one region would not impact SNS in another. The systems and services sometimes have cross-region communication capability for handling customer requested services, but each region's core operations for these regional services is independent.