Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Google Bigtable vs BigQuery for storing large number of events

Tags:

Background

We'd like to store our immutable events in a (preferably) managed service. Average size of one event is less than 1 Kb and we have between 1-5 events per second. The main reason for storing these events is to be able to replay them (perhaps using table scanning) once we create future services that might be interested in these events. Since we're in the Google Cloud we're obviously looking at Google's services as first choice.

I suspect that Bigtable would be a good fit for this but according to the price calculator it'll cost us more than 1400 USD per month (which to us is a big deal):

enter image description here

Looking at something like BigQuery renders a price of 3 USD per month (if I'm not missing something essential):

enter image description here

Even though a schema-less database would be better suited for us we would be fine with essentially storing our events as a blob with some metadata.

Questions

Could we use BigQuery for this instead of Bigtable to reduce costs? For example BigQuery has something called streaming inserts which to me seems like something we could use. Is there anything that'll bite us in the short or long term that I might not be aware of if going down this route?

like image 489
Johan Avatar asked Dec 23 '15 14:12

Johan


People also ask

What is difference between Google Bigtable and BigQuery?

Bigtable is a NoSQL wide-column database optimized for heavy reads and writes. On the other hand, BigQuery is an enterprise data warehouse for large amounts of relational structured data.

How much data can you store in BigQuery?

You can export up to 1 GB of table data to a single file. For larger tables, the results are written to multiple files. Use the BigQuery Storage Read API.

What is cloud Bigtable not good for?

Other storage and database options Bigtable is not a relational database. It does not support SQL queries, joins, or multi-row transactions. If you need full SQL support for an online transaction processing (OLTP) system, consider Cloud Spanner or Cloud SQL.

Is BigQuery used for Big Data?

BigQuery is suitable for “heavy” queries, those that operate using a big set of data. The bigger the dataset, the more you're likely to gain performance by using BigQuery.


1 Answers

Bigtable is great for large (>= 1TB) mutable data sets. It has low latency under load and is managed by Google. In your case, I think you're on the right track with BigQuery.

like image 168
Solomon Duskis Avatar answered Sep 27 '22 21:09

Solomon Duskis