BigQuery: Does bq load command support loading from named pipe as a source?

Question

I am trying to load data to Google bigquery using bq load from a named pipe.

Console Window1:

 $ mkfifo /usr/pipe1
 $ cat /dev1/item.dat > /usr/pipe1

Console Window2:

 $ bq load --source_format=CSV  projectid:dataset.itemtbl /usr/pipe1 field1:integer,field2:integer

Got the following error:

BigQuery error in load operation: Source path is not a file: /usr/pipe1

Answer 1

The BigQuery client bq.py does not support named pipes. It explicitly requires files:

https://code.google.com/p/google-bigquery-tools/source/browse/bq/bigquery_client.py?r=30df4638ff2ddb01d3f495af5c131ed3c2cfbd04#617

Allowing named pipes is a good feature suggestion. You can request it here:

https://code.google.com/p/google-bigquery/issues/list

It looks like you could tweak your copy of bigquery_client.py pretty easily to make this work as well. Good luck!