Webmasters API - Quota limits

Question

We're trying to download page data for sites using the Webmasters API .NET Client Library, by calling WebmastersService.SearchAnalytics.Query(). To do this we are using Batching and sending approx. 600 requests in one batch. However most of these fail with the error "Quota Exceeded". The amount that fail varies each time but it is only about 10 of the 600 that work (and it varies where they are within the batch). The only way we can get it to work is to reduce the batch size down to 3, and wait 1 second between each call.

According to the Developer Console our daily quota is set to 1,000,000 (and we have 99% remaining) and our per user limit is set to 10,000 requests / second / user.

The error we get back is:

Quota Exceeded [403] Errors [ Message[Quota Exceeded] Location[ - ] Reason[quotaExceeded] Domain[usageLimits]]

Is there another quota which is enforced? What does "Domain[usage limits]" mean - is the domain the site we are query the page data for, or is it our user account?

We still get the problem if we run each request separately, unless we wait 1 second between each call. Due to the number of sites and the number of pages we need to download the data for this isn't really an option.

I found this post which points out that just because the max batch size is 1000 doesn't mean to say the Google service you are calling supports batches of those sizes. But I'd really like to find out exactly what the quota limits really are (as they don't relate to the Developer Console figures) and how to avoid the errors.

Update 1

Here's some sample code. Its specially written just to prove the problem so no need to comment on it's quality ;o)

using Google.Apis.Auth.OAuth2;
using Google.Apis.Services;
using Google.Apis.Util.Store;
using Google.Apis.Webmasters.v3;
using Google.Apis.Webmasters.v3.Data;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            new Program().Run().Wait();
        }

        private async Task Run()
        {
            List<string> pageUrls = new List<string>();
            // Add your page urls to the list here

            await GetPageData("<your app name>", "2015-06-15", "2015-07-05", "web", "DESKTOP", "<your domain name>", pageUrls);
        }

        public static async Task<WebmastersService> GetService(string appName)
        {
            //if (_service != null)
            //    return _service;
            //TODO: - look at analytics code to see how to store JSON and refresh token and check runs on another PC
            UserCredential credential;
            using (var stream = new FileStream("c:\\temp\\WMT.json", FileMode.Open, FileAccess.Read))
            {
                credential = await GoogleWebAuthorizationBroker.AuthorizeAsync(
                    GoogleClientSecrets.Load(stream).Secrets,
                    new[] { Google.Apis.Webmasters.v3.WebmastersService.Scope.Webmasters },
                    "user", CancellationToken.None, new FileDataStore("WebmastersService"));
            }

            // Create the service.
            WebmastersService service = new WebmastersService(new BaseClientService.Initializer()
            {
                HttpClientInitializer = credential,
                ApplicationName = appName,
            });

            //_service = service;
            return service;

        }

        private static async Task<bool> GetPageData(string appName, string fromDate, string toDate, string searchType, string device, string siteUrl, List<string> pageUrls)
        {
            // Get the service from the initial method
            bool ret = false;

            WebmastersService service = await GetService(appName);
            Google.Apis.Requests.BatchRequest b = new Google.Apis.Requests.BatchRequest(service);

            try
            {
                foreach (string pageUrl in pageUrls)
                {
                    SearchAnalyticsQueryRequest qry = new SearchAnalyticsQueryRequest();
                    qry.StartDate = fromDate;
                    qry.EndDate = toDate;
                    qry.SearchType = searchType;
                    qry.RowLimit = 5000;
                    qry.Dimensions = new List<string>() { "query" };
                    qry.DimensionFilterGroups = new List<ApiDimensionFilterGroup>();
                    ApiDimensionFilterGroup filterGroup = new ApiDimensionFilterGroup();
                    ApiDimensionFilter filter = new ApiDimensionFilter();
                    filter.Dimension = "device";
                    filter.Expression = device;
                    filter.Operator__ = "equals";

                    ApiDimensionFilter filter2 = new ApiDimensionFilter();
                    filter2.Dimension = "page";
                    filter2.Expression = pageUrl;
                    filter2.Operator__ = "equals";

                    filterGroup.Filters = new List<ApiDimensionFilter>();
                    filterGroup.Filters.Add(filter);
                    filterGroup.Filters.Add(filter2);
                    qry.DimensionFilterGroups.Add(filterGroup);

                    var req = service.Searchanalytics.Query(qry, siteUrl);
                    b.Queue<SearchAnalyticsQueryResponse>(req, (response, error, i, message) =>
                    {
                        if (error == null)
                        {
                            // Process the results
                            ret = true;
                        }
                        else
                        {
                            Console.WriteLine(error.Message);
                        }

                    });
                    await b.ExecuteAsync();
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Exception occurred getting page  stats : " + ex.Message);
                ret = false;
            }

            return ret;
        }        
    }
}

Paste this into program.cs of a new console app and add Google.Apis.Webmasters.v3 via nuget. It looks for the wmt.json file in c:\temp but adjust the authentication code to suit your setup. If I add more than 5 page urls to the pageUrls list then I get the Quota Exceeded exception.

Answer 1

I've found that the stated quotas don't really seem to be the quotas. I had to slow my requests down to avoid this same issue (1/sec), even though I was always at or below the stated rate limit (20/sec). Furthermore, it claims that it gives a rateLimitExceeded error in the docs for going too fast, but really it returns a quotaExceeded error. It might have to do with how Google averages the rate of requests over time (as some of the requests we made were simultaneous, even though the long-run average was designed to be at or below 20/sec), but I cannot be sure.