How can I see the currently reached used quota in BQ?

Question

Google's BQ Quota policy indicates that there's a limit of 10,000 loads per project.

Under Google APIs website I can see the quotas for my project but the "Used" column is consistently at 0%. (Even though I know I reached the maximum)

How can I see the amount of load requests I performed in the recent day? Also, exactly at what hour does that quota reset?

Thanks.

Answer 1

There isn't a way to find out how many loads you've used in your per-day quota (other than listing your jobs in the last day and counting the number that were loads). We're considering adding more data to the quotas page in the Google Cloud Console page, we will add load limits to the list.

There are a couple of different types of quota limits in BigQuery. Some are hard limits that are designed to prevent users from taking too much of our finite capacity. The limit on concurrent 'large' queries is one of these, since large queries use a lot of resources and we don't want one customer to be able to be able to adversely affect performance for other users. Hard quotas tend to increase over time as we increase capacity.

There are other quotas that are 'soft' quotas, that are in place to encourage efficient usage of BigQuery. The load limit is one of these 'soft' quotas, that is designed to prevent users from doing lots of tiny updates or writing to lots of tiny tables.

If you're doing lots of small updates, you might be better off using the streaming insert mechanism (tabledata.insertAll()), which allows you to post rows to BigQuery that are immediately added to your tables. If you have lots of small load jobs that append to the same table, it means that the table will not be optimized until BigQuery's periodic coalesce operation runs on your table. Tables that are written to using tabledata.insertAll(), on the other hand, are written out in a much more efficient way, and don't need to be coalesced.

If you're writing to a lot of small tables, that may be an indication that you're planning to query over a large number of tables. Lots of small tables in a query means that BigQuery has to do a lot more work per query and has less flexibility in how it stores the data.

There are plenty of other reasons that you would want to have more than 10k loads per project per day. If you do, we'd like to understand your use case better. If you have a support arrangement with google cloud support (https://cloud.google.com/support/), your support contact can work with you on figuring out your needs and whether a quota bump makes sense.