Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to create a BigQuery dataset and a table/schema from Java client (without CSV file)

I think the method starting line 200 here is relevant (edit: I needed to add a parameter to the line Insert insertReq = bigquery.jobs().insert(PROJECT_ID, insertJob); ) but it doesn't work. I get "Load configuration must specify at least one source URI"

I have tried the following:

    TableSchema schema = new TableSchema();
    List<TableFieldSchema> tableFieldSchema = new ArrayList<TableFieldSchema>();
    TableFieldSchema schemaEntry = new TableFieldSchema();
    schemaEntry.setName(myFirstFieldName);
    schemaEntry.setType("STRING");
    tableFieldSchema.add(schemaEntry);
    schema.setFields(tableFieldSchema);

    Table table = new Table();
    table.setSchema(schema);
    table.setId(tableName);
    table.setCreationTime(System.currentTimeMillis());
    table.setKind("bigquery#table");
    try {
        bigquery.tables().insert(PROJECT_ID, DATASET_ID, table).execute();
    } catch (IOException e) {
    }

but I get an error Required parameter is missing

like image 965
user1176505 Avatar asked Jan 17 '14 10:01

user1176505


2 Answers

OK based on the idea by Jordan Tigani, here is the Java code that works to create a blank table in BigQuery with Google Java API Client:

TableSchema schema = new TableSchema();
List<TableFieldSchema> tableFieldSchema = new ArrayList<TableFieldSchema>();
TableFieldSchema schemaEntry = new TableFieldSchema();
schemaEntry.setName(myFirstFieldName);
schemaEntry.setType("STRING");
tableFieldSchema.add(schemaEntry);
schema.setFields(tableFieldSchema);

Table table = new Table();
table.setSchema(schema);
TableReference tableRef = new TableReference();
tableRef.setDatasetId(DATASET_ID);
tableRef.setProjectId(PROJECT_ID);
tableRef.setTableId(tableId);
table.setTableReference(tableRef);
try {
    bigquery.tables().insert(PROJECT_ID, DATASET_ID, table).execute();
} catch (IOException e) {
}

To create a dataset (before creating the table)

    Dataset dataset = new Dataset();
    DatasetReference datasetRef = new DatasetReference();
    datasetRef.setProjectId(PROJECT_ID);
    datasetRef.setDatasetId(DATASET_ID);
    dataset.setDatasetReference(datasetRef);
    try {
        bigquery.datasets().insert(PROJECT_ID, dataset).execute();
    } catch (IOException e) {
    }
like image 81
user1176505 Avatar answered Sep 25 '22 13:09

user1176505


Try setting the project id and the dataset id on the table (I realize that it seems redundant becuase you specify them on the insert() operation, but that is a quirk of REST ... the project and dataset are part of the URL, but they are also part of the resource.

From a raw HTTP api level, the following works:

https://www.googleapis.com/bigquery/v2/projects/myproject/datasets/mydataset/tables?alt=json    
{"tableReference": 
    {"tableId": "dfdlkfjx", "projectId": "myproject", "datasetId": "mydataset"},
 "schema": 
     {"fields": [{"name": "a", "type": "STRING"}]}}
like image 42
Jordan Tigani Avatar answered Sep 25 '22 13:09

Jordan Tigani