Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Adding a variable from python environment to queries when using BigQuery Jupyter magic command

I have a few cells in my Jupyter notebook that is querying data from Google's BigQuery. Parts of these queries share the same variables. I am looking for the cleanest way to factor these variables so I don't have to manually set them individually every time I want to make a change.

Here is an example of what I have:

%%bigquery df
select * from `project.dataset.table`
where date_str like '2019-02-%'

Here is what I am looking for:

date_str_query = '2019-02-%' # this variable would be in the python environment

%%bigquery df
select * from `project.dataset.table`
where date_str like date_str_query
like image 865
sinapan Avatar asked Apr 17 '19 15:04

sinapan


People also ask

How do you declare a variable in a large query?

SET (a, b, c) = (1 + 3, 'foo', false); The following example assigns the result of a query to multiple variables. First, it declares two variables, target_word and corpus_count ; next, it assigns the results of a SELECT AS STRUCT query to the two variables.


1 Answers

##### UPDATE #####

New documentation has a detailed description of various inputs to the magic command. It can be found here: https://googleapis.github.io/google-cloud-python/latest/bigquery/magics.html?highlight=gender

##### ORIGINAL POST #####

Unfortunately, this part is missing from the official documentation, and I had to scrape through the code to finally find a solution. I'm documenting my findings here in hopes to help people with the same issue. There are two ways to do this.

Way 1 [not using the magic command at all]:

date_str_query = '2019-02-%'
sql = """
select * from `project.dataset.table`
where date_str like {0}
""".format(date_str_query)
df = client.query(sql).to_dataframe()

Way 2 [magic command rules!]:

params = {'date_str_query': '%-2019-02-%'}

%%bigquery df --params $params
select * from `project.dataset.table`
where date_str like @date_str_query

I prefer Way 2 because you get SQL syntax highlighting and a bunch of other features with the magic command.

References:

Google's Documentation: https://cloud.google.com/bigquery/docs/visualize-jupyter

The source code: https://github.com/googleapis/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/magics.py

like image 127
sinapan Avatar answered Oct 22 '22 05:10

sinapan