Google BigQuery local emulation

Question

Is there any local server implementation for google BigQuery like localstack for AWS.

There is a reference to local server implementation for app engine with dev_appserver.py. I could run this with a dummy app.yaml.

• I can not access interactive console using localhost:8000/console
• Can I run bq command line utility after starting server using dev_appserver.py --enable-console app.yaml

Answer 1

While there is no emulator provided by Google, there is an open source BigQuery emulator available https://github.com/goccy/bigquery-emulator

I'm using this, and quite useful for local development and unit testing.

To run BQ emulator

$ docker pull ghcr.io/goccy/bigquery-emulator:latest --project=test-local
[bigquery-emulator] REST server listening at 0.0.0.0:9050
[bigquery-emulator] gRPC server listening at 0.0.0.0:9060

If you are using Go, you need to pass following options to new bq client

client, err := bigquery.NewClient(
    ctx,
    projectID,
    option.WithEndpoint("http://0.0.0.0:9050"),
    option.WithoutAuthentication(),
  )

BigQuery emulator utilizes SQLite for storage. You can select either memory or file as the data storage destination at startup, and if you set it to file, data can be persisted.

You can load seeds from a YAML file on startup

There is also an open ticket for Google to create emulator for BQ https://issuetracker.google.com/issues/129248927