Frequent RESOURCE_EXHAUSTED errors

Question

I'm getting into the Google Cloud Deployment Manager lately but struggling with some strange errors. Maybe someone here will have some idea as to the cause.

Typically I will "create" a deployment in preview mode, then once I see it in the GUI I will click the "Deploy" button and have it do it's thing. But when it finishes I am greeted with the following error message:

{
    "ResourceType": "deploymentmanager.v2.virtual.enableService",
    "ResourceErrorCode": "429",
    "ResourceErrorMessage": {
        "code": 429,
        "message": "Insufficient tokens for quota 'DefaultGroup' and limit 'CLIENT_PROJECT-100s' of service 'servicemanagement.googleapis.com' for consumer 'project_number:794362051698'.",
        "status": "RESOURCE_EXHAUSTED",
        "details": [{
            "@type": "type.googleapis.com/google.rpc.Help",
            "links": [{
                "description": "Google developer console API key",
                "url": "https://console.developers.google.com/project/794362051698/apiui/credential"
            }]
        }],
        "statusMessage": "Too Many Requests",
        "requestPath": "https://servicemanagement.googleapis.com/v1/operations/projectSettings.d51a6a47-7a15-42bb-90a7-70c3dd7c25e3"
    }
}

One might think this is due to a quota being exceeded, but after checking the quotas in IAM->Quotas I can't find anything over quota.

There are now 3 deployments piled up in my Deployment Manager in a similar sorry state. The weird thing is that my project was created and when I look at the enabled APIs for that project I find everything that I specified in my DM configuration.

Edit: The problem seems to go away when I remove some of the APIs which I enable (using the apis: block in my resource: block). I'm trying to enable 23 but the problem goes away when I cut that number in half.

Answer 1

This issue is discussed in https://github.com/GoogleCloudPlatform/google-cloud-node/issues/1756 . The upshot is:

I was giving each subscription a unique subscription name (uuid) to ensure messages arrive to a single subscriber at a time, so one way to reduce usage is to avoid that and instead ignore irrelevant messages. But the main thing that solved it was increasing the quota in the developer console if I remember correctly the default quota is 1K, I took it up to the maximum (10K); it wasn't clear you could actually do that from the console. Finally, I'm now using a different messaging queue for few reasons one of which I I thought 10K was too little - that did it ;)