How to Sync Mysql into Bigquery in realtime?

Question

Currently I have some script which first deletes the table and upload the table from MySQL to Bigquery. And many time it had failed. Plus it run only once a day. I am looking for some scalable and realtime solution. Your Help will be much appreciated :)

Answer 1

Read these series of posts from Wepay, where they detail how they sync their MySQL databases to BigQuery, using Airflow:

• https://wecode.wepay.com/posts/wepays-data-warehouse-bigquery-airflow
• https://wecode.wepay.com/posts/airflow-wepay
• (3rd one is about BigQuery)

As a summary (quoting):

• Setup authentication, connections, DAG.
• Define which columns to pull from MySQL and load into BigQuery.
• Choose how to load the data: incrementally, or fully.
• De-duplicating.