Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

BiqQuery command line load query - using a JSON table schema

I'm trying to load a comma separated test file, 'Davetest.csv', and use a JSON schema held in the file 'testjson.json'

However I can't get it to recognize the table schema parameter properly.

load mydataset.testy gs://paps1/davetest.csv gs://paps1/testjson.json

gives me

BigQuery error in load operation: Invalid value for: //PAPS1/TESTJSON.JSON is not a valid value

I tried it without the path for the table schema (i.e.load mydataset.testy gs://paps1/davetest.csv testjson.json), and this confused it into thinking I was trying to write the schema inline with a first fieldname of 'testjson.json', and so failed on that as well.

Both of the files are held on GCS in the same bucket.

Obviously I'm doing something fundamentally wrong, but all the code examples I've seen are using the structure above, as far as I can tell.

Could anyone advise?

like image 431
Dave Papworth Avatar asked Jan 01 '26 02:01

Dave Papworth


1 Answers

I think documentation for "bq Command-Line Tool" is not clear enough! https://cloud.google.com/bigquery/bq-command-line-tool#creatingtablefromfile

Rather, check examples by running > bq help load
They are more specific in showing allowed options for schema:

Examples:  
bq load ds.new_tbl ./info.csv ./info_schema.json  
bq load ds.new_tbl gs://mybucket/info.csv ./info_schema.json  
bq load ds.small gs://mybucket/small.csv name:integer,value:string  
bq load ds.small gs://mybucket/small.csv field1,field2,field3  

As you can see now, schema file is expected to be local to bq tool

And, of course using inline text schema is still option if local file is somehow doesnt fit in your design

like image 53
Mikhail Berlyant Avatar answered Jan 03 '26 13:01

Mikhail Berlyant



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!