Copy data from AWS S3 to Aurora Postgres

Question

I'm trying to copy data from AWS S3 to Aurora Postgres. Currently my process is as follows:

1. Download files from S3 locally (on EC2 instance)
2. Run "COPY FROM STDIN ... " command to load data from input stream to Aurora postgres.

I'm wondering if there is a command to copy directly from S3 to Aurora postgres.

Answer 1

This blog post from AWS mentions being able to do it with AWS Database Migration service

https://aws.amazon.com/blogs/database/stream-data-into-an-aurora-postgresql-database-using-aws-dms-and-amazon-kinesis-data-firehose/

AWS DMS can read data from source S3 buckets and load them into a target database. To do this, provide access to an S3 bucket containing one or more data files. In that S3 bucket, include a JSON file that describes the mapping between the data and the database tables of the data in those files.

The source data files must be in comma-separated value (CSV) format. Name the files using the naming convention shown following. In this convention, schemaName is the source schema and tableName is the name of a table within that schema.