Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Google Bigquery query execution using google cloud dataflow

Is it possible to execute Bigquery's query using Google cloud data flow directly and fetch data, not reading data from table then putting conditions?

For example, PCollections res=p.apply(BigqueryIO.execute("Select col1,col2 from publicdata:samples.shakeseare where ...."))

Instead of reinventing using iterative method what Bigquery queries already implemented, we can use the same directly.

Thanks and Regards

Ajay K N

like image 403
Ajay K N Avatar asked Aug 11 '26 10:08

Ajay K N


1 Answers

BigQueryIO currently only supports reading from a Table and not a Query or View (FAQ).

One way to work around this is in your main program to create a BigQuery permanent table by issuing a query before you run your Dataflow job. After, your job runs you could delete the table.

like image 153
Jeremy Lewi Avatar answered Aug 14 '26 11:08

Jeremy Lewi



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!