BigQuery Results to Panda DataFrame in Chunks

Question

I am trying to save the results of a BigQuery query to a Panda DataFrame using bigquery.Client.query.to_dataframe()

This query can return millions of rows.

Given that Panda to BQ (Dataframe.to_gbq()) has a chunk parameter, is there something similar for BQ to Pandas to incrementally add to the dataframe without having to run the query multiple times with a limit and offset?

Answer 1

You can use to_dataframe_iterable instead to do this.

job = client.query(query)
result = job.result(page_size=20)

for df in result.to_dataframe_iterable():
    # df will have at most 20 rows
    print(df)