Stream a large Big-query SELECT with Node?

Question

I'm developing a Node.js program. We use Node to manipulate data every day that are store in Big Query.

Each day we have a high volume of new data (280 Go).

How to make a request on BigQuery on all the day and stream the result row after row ?

Now, we don't have stream. We just request all the data once.

I could use the sql LIMIT keyword. But the problem is that BigQuery ignore the LIMIT in cost calculation. If we LIMIT 0,10. It explores all the data of the day (280 Go). Idem for LIMIT 10,10 ...

This is my current code.

    const BigQuery = require('@google-cloud/bigquery');

    // ... Some code ...

    this.bigQuery
        .query(Exporter.enrichQueryWithOptions(`SELECT e.name FROM  events))
        .then(results => {
            const rows = results[0];
            console.log(rows);
        })
        .catch(err => {
            console.error('ERROR:', err);
        });

Answer 1

I think this might be what you need:

https://googleapis.dev/nodejs/bigquery/latest/BigQuery.html#createQueryStream

That function allows you to build a query and consume it through a stream of data.