Background
We'd like to store our immutable events in a (preferably) managed service. Average size of one event is less than 1 Kb and we have between 1-5 events per second. The main reason for storing these events is to be able to replay them (perhaps using table scanning) once we create future services that might be interested in these events. Since we're in the Google Cloud we're obviously looking at Google's services as first choice.
I suspect that Bigtable would be a good fit for this but according to the price calculator it'll cost us more than 1400 USD per month (which to us is a big deal):
Looking at something like BigQuery renders a price of 3 USD per month (if I'm not missing something essential):
Even though a schema-less database would be better suited for us we would be fine with essentially storing our events as a blob with some metadata.
Questions
Could we use BigQuery for this instead of Bigtable to reduce costs? For example BigQuery has something called streaming inserts which to me seems like something we could use. Is there anything that'll bite us in the short or long term that I might not be aware of if going down this route?
Bigtable is a NoSQL wide-column database optimized for heavy reads and writes. On the other hand, BigQuery is an enterprise data warehouse for large amounts of relational structured data.
You can export up to 1 GB of table data to a single file. For larger tables, the results are written to multiple files. Use the BigQuery Storage Read API.
Other storage and database options Bigtable is not a relational database. It does not support SQL queries, joins, or multi-row transactions. If you need full SQL support for an online transaction processing (OLTP) system, consider Cloud Spanner or Cloud SQL.
BigQuery is suitable for “heavy” queries, those that operate using a big set of data. The bigger the dataset, the more you're likely to gain performance by using BigQuery.
Bigtable is great for large (>= 1TB) mutable data sets. It has low latency under load and is managed by Google. In your case, I think you're on the right track with BigQuery.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With