Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

scheduled export from Google BigQuery to Google Cloud Storage

I'm new to Google Cloud and would like to know best use cases on how to schedule queries and export them to Google Cloud Storage. I've seen documentations on how to manually export data but couldn't find anything specific on doing it in an automated way. Is there any best way on how to approach this ?

Thanks

like image 614
RoS Avatar asked Nov 04 '17 14:11

RoS


1 Answers

  1. Implement your table export function [1] using Node.js, python or Go. These languages are supported by Cloud Functions and BigQuery.
  2. Insert the above function in Cloud Function [2] service which is an event driven serverless compute platform.
  3. Trigger the above function using Cloud Scheduler [3] using a schedule interval of your choice. The schedule interval can be specified using a cron expression. The scheduler will trigger the function via a REST call on the function´s endpoint.
  4. Verify the success of the above operation by visiting your bucket and ensuring that the table(s) has been successfully exported.
like image 105
Moe Avatar answered Oct 01 '22 04:10

Moe