Big query EXPORT DATA statement creating mutiple files with no data and just header record

Question

I have read similar issue here but not able to understand if this is fixed.

Google bigquery export table to multiple files in Google Cloud storage and sometimes one single file

I am using below big query EXPORT DATA OPTIONS to export the data from 2 tables in a file. I have written select query for the same.

EXPORT DATA OPTIONS(
uri='gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_'||CURRENT_DATE()||'*.csv',
format='CSV',
overwrite=true,
header=true,
field_delimiter='|') AS     
SELECT

I have only 2 rows returning from my select query and I assume that only one file should be getting created in google cloud storage. Multiple files are created only when data is more than 1 GB. thats what I understand.

However, 3 files got created in cloud storage where 2 files just had the header record and the third file has 3 records(one header and 2 actual data record)

radhika_sharma_ibm@cloudshell:~ (whr-asia-datalake-nonprod)$ gsutil ls gs://whr-asia-datalake-dev-standard/outbound/Adobe/
gs://whr-asia-datalake-dev-standard/outbound/Adobe/
gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_2021-02-04000000000000.csv
gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_2021-02-04000000000001.csv
gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_2021-02-04000000000002.csv

Why empty files are getting created? Can anyone please help? We don't want to create empty files. I believe only one file should be created when it is 1 GB. more than 1 GB, we should have multiple files but NOT empty.

Answer 1

You have to force all data to be loaded into one worker. In this way you will be exporting only one file (if <1Gb). My workaround: add a select distinct * on top of the Select statement.