How to read BigQuery table using python pipeline code in GCP Dataflow

Question

Could someone please share syntax to read/write bigquery table in a pipeline written in python for GCP Dataflow

Answer 1

Run on Dataflow

First, construct a Pipeline with the following options for it to run on GCP DataFlow:

import apache_beam as beam

options = {'project': <project>,
           'runner': 'DataflowRunner',
           'region': <region>,
           'setup_file': <setup.py file>}
pipeline_options = beam.pipeline.PipelineOptions(flags=[], **options)
pipeline = beam.Pipeline(options = pipeline_options)

Read from BigQuery

Define a BigQuerySource with your query and use beam.io.Read to read data from BQ:

BQ_source = beam.io.BigQuerySource(query = <query>)
BQ_data = pipeline | beam.io.Read(BQ_source)

Write to BigQuery

There are two options to write to bigquery:

• use a BigQuerySink and beam.io.Write:

  BQ_sink = beam.io.BigQuerySink(<table>, dataset=<dataset>, project=<project>)
  BQ_data | beam.io.Write(BQ_sink)
  

• use beam.io.WriteToBigQuery:

  BQ_data | beam.io.WriteToBigQuery(<table>, dataset=<dataset>, project=<project>)