Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Realtime Analytics with BigQuery

Is there a way to run real time analytics with BigQuery? I used the CSV upload option that kicks off a job and loads the data in an offline mode which can be analyzed once the load completes. But in the announcement regarding BigQuery there are mentions of using BigQuery for realtime analytics. How can this be achieved? Can we append (no updates) data from Google Cloud database to BigQuery in a trickle mode for realtime analytics?

As a sidenote, I noticed that BigQuery CSV data loads are about an order of magnitude slower than LucidDB and InfiniDB running on my local PC using a 10GB data file. It took 34 minutes for the BigQuery job to complete vs. 5 minutes on InfiniDB and LucidDB. The query execution times (for simple aggregates) are twice as slower on BigQuery compared to InfiniDB (6 seconds vs. 3 seconds against the 10GB file loaded with approx. 30+ Million records) but better than LucidDB.

like image 529
invertigo Avatar asked Sep 09 '26 12:09

invertigo


1 Answers

2013 update: Now you can stream data into BigQuery, and query it in realtime.

(data you stream in is available instants later for querying)

https://developers.google.com/bigquery/streaming-data-into-bigquery

like image 121
Felipe Hoffa Avatar answered Sep 11 '26 13:09

Felipe Hoffa