Menu
I'm investigating the merits of using BigQuery for gaining insights into application logfiles. The logs are produced by Java and C# applications, most of them on cloud-based VMs. I'm interested to hear whether others have done this and of the relative merits of BigQuery vs ElasticSearch/Logstash/Kibana.
The advantage of BigQuery seems to be that it can deal with huge amounts of data whereas the ELK solutions seems maybe better suited to the non-structured nature of logfiles, especially when they come from different systems.
I'd also like to display information on a dashboard. Kibana seems to be very good for that. How easy is it to create dashboards using the Google solution (using google sheets, etc)?
Thoughts, use-cases?
791
BigQuery is a fully managed enterprise data warehouse that helps you manage and analyze your data with built-in features like machine learning, geospatial analysis, and business intelligence.
Data is stored in a columnar storage fashion which makes possible to achieve a very high compression ratio and scan throughput. Tree Architecture is used for dispatching queries and aggregating results across thousands of machines in a few seconds.
Use it when you have queries that run more than five seconds in a relational database. The idea of BigQuery is running complex analytical queries, which means there is no point in running queries that are doing simple aggregation or filtering.
2017 update: Elastic officially supported on GCP
Elasticsearch and BigQuery work great together. BigQuery will take as much data as you have and query it in any way you want in seconds. Meanwhile a well tuned Elasticsearch installation will give you answers in less than a second, but only for certain queries over a limited amount of data.
See this post by Ory at Rounds, where they detail how they use both:
https://medium.com/@oryband/collecting-user-data-and-usage-ffa84c4dba34
The two top titles that summarize their reasons to do both:
154
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
