Getting "Query too large" in BigQuery

Question

I am storing event data in BigQuery, partitioned by day - one table per day. The following query failed:

select count(distinct event) 
from TABLE_DATE_RANGE(my_dataset.my_dataset_events_, SEC_TO_TIMESTAMP(1391212800), SEC_TO_TIMESTAMP(1393631999))

Each table is about 8GB in size.

Has anyone else experienced this error? Seems like it's limited by table size, because in this query, I've only limited it to just one column. When I use a smaller time range, it works.. but the whole point of using BigQuery was its support for large datasets.

Answer 1

"Query too large" in this case means that the TABLE_RANGE is getting expanded internally to too many tables, generating an internal query that is too large to be processed.

This has 2 workarounds:

• Query less tables (could you aggregate these tables into a bigger one?).
• Wait until the BQ team solves this issue internally. Instead of using a workaround, you should be able to run this query unchanged. Just not today :).