Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Getting results as JSON from BigQuery with google-cloud-python

I'm querying BigQuery via google-cloud-python as follows:

client = bigquery.Client()

query = """SELECT * FROM `{dataset}.{table}`
  WHERE id=@id LIMIT 1""".format(dataset=dataset,
                                 table=table)

param = ScalarQueryParameter('id', 'STRING', id)
query = client.run_sync_query(query, query_parameters=[param])
query.use_legacy_sql = False
query.timeout_ms = 1000
query.run()

assert query.complete

try:
    results = query.rows[0]
except IndexError:
    results = None

This returns data like:

[
    "Tue, 11 Apr 2017 03:18:52 GMT",
    "A132",
    "United Kingdom",
    [
        {
            "endDate": "2012-12-05",
            "startDate": "2011-12-27",
            "statusCode": "Terminated"
        }
    ]
]

The repeated field has been converted to JSON. But I want the rest of the data to be converted to JSON as well. I could implement this myself by examining query.schema but it seems like this should be in the library since it already happens for repeated elements.

How can I get BigQuery query results formatted as JSON using this library? E.g.:

{
    "timestamp": "Tue, 11 Apr 2017 03:18:52 GMT",
    "id": "A132",
    "country": "United Kingdom",
    [
        {
            "endDate": "2012-12-05",
            "startDate": "2011-12-27",
            "statusCode": "Terminated"
        }
    ]
}
like image 486
jbrown Avatar asked Apr 12 '17 15:04

jbrown


People also ask

How extract JSON data from BigQuery?

JSON_EXTRACT. Extracts a JSON value, such as an array or object, or a JSON scalar value, such as a string, number, or boolean. If a JSON key uses invalid JSONPath characters, then you can escape those characters using single quotes and brackets. Extracts a JSON null when a JSON null is encountered.

How do you export results from BigQuery?

Open the BigQuery page in the Google Cloud console. In the Explorer panel, expand your project and dataset, then select the table. In the details panel, click Export and select Export to Cloud Storage.

Can BigQuery store JSON?

BigQuery natively supports JSON data using the JSON data type. This document describes how to create a table with a JSON column, insert JSON data into a BigQuery table, and query JSON data.


1 Answers

As it turns out, the code is simple enough:

field_names = [f.name for f in query.schema]

try:
    raw_results = query.rows[0]
    zipped_results = zip(field_names, raw_results)
    results = {x[0]: x[1] for x in zipped_results}
except IndexError:
    results = None
like image 164
jbrown Avatar answered Oct 06 '22 03:10

jbrown