Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

BigQuery Date-Partitioned Views

BigQuery allows you to create date-partitioned tables: https://cloud.google.com/bigquery/docs/creating-partitioned-tables

I'd like to be able to create views on top of date-partitioned tables and ideally still reap the performance benefits. Everything I've read suggests that this isn't possible?

Has anyone had any luck otherwise?

like image 818
tomwoodruff Avatar asked Jun 27 '17 13:06

tomwoodruff


People also ask

Can view be partitioned in BigQuery?

If a query uses a qualifying filter on the value of the partitioning column, BigQuery can scan the partitions that match the filter and skip the remaining partitions. This process is called partition pruning. Partition pruning is the mechanism BigQuery uses to eliminate unnecessary partitions from the input scan.

Can views be partitioned?

A partitioned view is a view defined by a UNION ALL of member tables structured in the same way, but stored separately as multiple tables in either the same instance of SQL Server or in a group of autonomous instances of SQL Server servers, called federated database servers.


2 Answers

Define your view to expose the partitioning pseudocolumn, like this:

SELECT *, EXTRACT(DATE FROM _PARTITIONTIME) AS date
FROM Date partitioned table;

Now if you query the view using a filter on date, it will restrict the partitions that are read.

like image 91
Elliott Brossard Avatar answered Oct 02 '22 12:10

Elliott Brossard


For anyone trying to do this with a wildcard partition date table such as Firebase or Google Analytics:

create view some.view as ( 
    select *, _TABLE_SUFFIX as suffix from 
    `firebase-public-project.analytics_153293282.events_*` 
)


 select * from some.view WHERE suffix = '20180814'
like image 41
ernesto Avatar answered Oct 02 '22 10:10

ernesto