Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Amazon RedShift Copy Command

I am using the data found on this page to practice the copy command.

But i am getting an exception

org.postgresql.util.PSQLException: ERROR: S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect,Rid 8D10A2C0B9C7570E,ExtRid VTQZsFdP8DRiJPza+Ko4bc2Y91P9Wra0Qb9C
  Detail: 
  -----------------------------------------------
  error:  S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect,Rid 8D10A2C0B9C7570E,ExtRid VTQZsFdP8DRiJPza+Ko4bc2Y91P9Wra0Qb9C
  code:      8001
  context:   Listing bucket=awssampledb prefix=tickit/allusers_pipe.txt
  query:     576
  location:  s3_utility.cpp:525
  process:   padbmaster [pid=4283]

What am i doing wrong ?

like image 990
AndroidDev Avatar asked Jul 21 '14 08:07

AndroidDev


People also ask

How do I use AWS Copy command?

The COPY command needs authorization to access data in another AWS resource, including in Amazon S3, Amazon EMR, Amazon DynamoDB, and Amazon EC2. You can provide that authorization by referencing an IAM role that is attached to your cluster or by providing the access key ID and secret access key for an IAM user.

How do I transfer data from Redshift to S3 using COPY command?

The syntax to specify the files to be loaded by using a manifest file is as follows: copy <table_name> from 's3://<bucket_name>/<manifest_file>' authorization manifest; The table to be loaded must already exist in the database. For information about creating a table, see CREATE TABLE in the SQL Reference.


2 Answers

The S3 bucket and redshift must be in the same region (US East):

This example uses an Amazon S3 bucket that is located in the US East (Northern Virginia) region. When you load data using a COPY command, the bucket containing your data must be in the same region as your cluster.

If your redshift cluster is in another region, look at Step 5: Load Sample Data from Amazon S3, scrolling to the table with region-specific-bucket-name:

------------------------------+-------------------------------+
| Region                      |  region-specific-bucket-name  |
------------------------------+-------------------------------+
| US East (Northern Virginia) |  awssampledb                  |
------------------------------+-------------------------------+
| US West (Oregon)            |  awssampledbuswest2           |
------------------------------+-------------------------------+
| EU (Ireland)                |  awssampledbeuwest1           |
------------------------------+-------------------------------+
| Asia Pacific (Singapore)    |  awssampledbapsoutheast1      |
------------------------------+-------------------------------+
| Asia Pacific (Sydney)       |  awssampledbapsoutheast2      |
------------------------------+-------------------------------+
| Asia Pacific (Tokyo)        |  awssampledbapnortheast1      |
------------------------------+-------------------------------+
like image 170
Tomasz Tybulewicz Avatar answered Nov 12 '22 01:11

Tomasz Tybulewicz


Specify the source region as below

copy pep.test 
(operation, task_id, task_name)
from 's3://move-redshift-data-devo/move-redshift-data-devo/'
iam_role 'arn:aws:iam::355750824903:role/RedshiftDynamoDBAccess'
format as csv
region as 'us-east-1';

https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-data-source-s3.html#copy-region

like image 32
swarnim gupta Avatar answered Nov 12 '22 00:11

swarnim gupta