Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to convert results returned from bigquery to Json format using Python?

Using Python to select data from Bigquery public dataset, after getting the result need to print it in JSON format.How to convert the result to JSON ? Thanks!

Have tried row[0] but errors out.

try:
    raw_results = query.rows[0]
    zipped_results = zip(field_names, raw_results)
    results = {x[0]: x[1] for x in zipped_results}
except IndexError:
    results = None

# from google.cloud import bigquery
# client = bigquery.Client()

query = """
    SELECT word, word_count
    FROM `bigquery-public-data.samples.shakespeare`
    WHERE corpus = @corpus
    AND word_count >= @min_word_count
    ORDER BY word_count DESC;
"""
query_params = [
    bigquery.ScalarQueryParameter("corpus", "STRING", "romeoandjuliet"),
    bigquery.ScalarQueryParameter("min_word_count", "INT64", 250),
]
job_config = bigquery.QueryJobConfig()
job_config.query_parameters = query_params
query_job = client.query(
    query,
    # Location must match that of the dataset(s) referenced in the 
    query.location="US",
    job_config=job_config,
)  # API request - starts the query

# Print the results
for row in query_job:
    print("{}: \t{}".format(row.word, row.word_count))
assert query_job.state == "DONE"
like image 596
Qianmei Avatar asked Apr 15 '19 00:04

Qianmei


People also ask

How do I export BigQuery results?

Open the BigQuery page in the Google Cloud console. In the Explorer panel, expand your project and dataset, then select the table. In the details panel, click Export and select Export to Cloud Storage.

Can BigQuery store JSON?

BigQuery natively supports JSON data using the JSON data type. This document describes how to create a table with a JSON column, insert JSON data into a BigQuery table, and query JSON data.


2 Answers

There is no current method for automatic conversion, but there is a pretty simple manual method to convert to json:

records = [dict(row) for row in query_job]
json_obj = json.dumps(str(records))

Another option is to convert using pandas:

df = query_job.to_dataframe()
json_obj = df.to_json(orient='records')
like image 197
Meow Avatar answered Nov 12 '22 04:11

Meow


You can actually just have BigQuery produce JSON directly. Change your query like this:

query = """
SELECT TO_JSON_STRING(word, word_count) AS json
FROM `bigquery-public-data.samples.shakespeare`
WHERE corpus = @corpus
AND word_count >= @min_word_count
ORDER BY word_count DESC;
"""

Now the result will have a single column named json with JSON-formatted output.

like image 35
Elliott Brossard Avatar answered Nov 12 '22 04:11

Elliott Brossard