Synchronize data from MySql to Amazon RedShift

Question

We do some aggregation on huge datasets in Amazon RedShift, and we have some relatively small amount of data in MySQL. For some of the joins in RedShift we need the data in MySQL. What is the best way to synchronize the MySql data to RedShift? Is there such a thing in redshift like the remote view in oracle? Or should I programatically query MySql and insert / update in RedShift?

Answer 1

Redshift now supports loading data from remote hosts via SSH. This technique involves:

1. Adding the public key from the cluster to the authorized_keys file on the remote host(s)
2. Allowing SSH access to the remote host(s) from the IP addresses of the cluster nodes
3. Uploading a JSON manifest to S3 specifying the remote host(s), the public key(s), and the command(s) to execute on the remote host
4. Running the COPY command with a specified manifest file and AWS credentials

The command specified by the manifest runs an arbitrary command that prints text output in a format suitable for ingest by the Redshift COPY command.

Answer 2

When MySQL data is required for joins in Redshift, we usually just send it over from one to another.

It implies:

1. Redshift: Creating an analogous table schema (bearing in mind Redshift/PSQL's particularities)
2. MySQL: Dumping the data table (in csv format)
3. Zipping the export, and sending it to S3
4. Redshift: Truncating the table, and importing all data using COPY

Steps 2 to 4 can be scripted, and allow you to send fresh data over to Redshift when necessary or regularly.