When should I prefer batch analysis over interactive analysis?

Question

The incentive to use batch queries instead of interactive mode queries was pricing, but with newer price changes there is no cost difference anymore - so is there any other incentive (quota, performance, other...) to use batch queries?

Answer 1

With the price change, there are two primary reasons to use batch priority:

• it lets you queue up your jobs.
• it lets you run low priority queries in a way that doesn't impact high priority ones.

There are a number of rate limits that affect interactive (i.e. non-batch) queries -- you can have at most 20 running concurrently, there are concurrent byte limits and 'large query' limits. If those limits are hit, the query will fail immediately. This is because BigQuery assumes that an interactive query is something you need run immediately.

When you use batch, if you ever hit a rate limit, the query will be queued and retried later. There are still similar rate limits, but they operate separately from interactive rate limits, so your batch queries won't affect your interactive ones.

One example might be that you have periodic queries that you run daily or hourly to build dashboards. Maybe you have 100 queries that you want to run. If you try to run them all at once as interactive, some will fail because of concurrent rate limits. Additionally, you don't necessarily want these queries to interfere with other queries you are running manually from the BigQuery Web UI. So you can run the dashboard queries at batch priority and the other queries will run normally as interactive.

One other point to note is that the scheduling for Batch queries has changed so the average wait times should come down considerably. Instead of waiting a half hour or so, batch queries should start within a minute or two (subject to queueing, etc).