CosmosDb with DocumentDB API through Azure API Management

Question

I am attempting to query my new CosmosDB collection through API management. Once proved out, this will be a front-end for user access to logged data. For that reason, I have the data partitioned by subscription ID. In Azure Portal for the Logs collection of my WebApi DB I see partition key as /api_subscription_key. I have the data going in from API Mgt. -> Event Hub -> Stream Analytics -> Cosmos.

Using the query explorer in Azure portal, I can try a query like:

SELECT * FROM c WHERE c.api_subscription_key = '573a1c65bceb52192c140131'

this brings back expected documents I have been successfully writing to CosmosDB for many days

    [
      {
        "eventenqueuedutctimesecond": "2017-07-27T15:09:02Z",
        "business_unit_key": null,
        "user_key": null,
        "api_message_id": "1718ea66-d225-45ec-b3fc-5daff4c7f426",
        "api_identifier": "21926e9d-9206-42b0-b4b1-7e7f1eb4e7dd",
        "api_id": "58d94cc622be39392343d4b6",
        "api_operation_id": "58e682bde055cd0ba4215d4b",
        "api_adapter_id": "573a1c64bceb520aac127ee5",
        "api_subscription_id": "573a1c65bceb52192c140131",
        "api_policy_id": "64BC4270-54AC-42DA-835C-E285F35BCA81",
        "basic_username": "",
        "message_version": "10",
        "claim_business_unit_key": null,
        "claim_user_key": null,
    ...
        "lasterrorsource": null,
        "lasterrorreason": null,
        "lasterrorscope": null,
        "lasterrorsection": null,
        "lasterrorpolicyid": null,
        "id": "7/27/2017 3:09:02 PM",
        "_rid": "9Fc0ANW4fwAoAAAAAAAADA==",
        "_self": "dbs/9Fc0AA==/colls/9Fc0ANW4fwA=/docs/9Fc0ANW4fwAoAAAAAAAADA==/",
        "_etag": "\"0700d90c-0000-0000-0000-597a020e0000\"",
        "_attachments": "attachments/",
        "_ts": 1501168140
      }...

My CosmosDB instance is plexconnectcosmos. Thru API management and its policies I am POSTing to

https://plexconnectcosmos.documents.azure.com/dbs/WebApi/colls/Logs/docs

with these headers (many vestigial and hopefully having no effect):

[
{
name: "Postman-Token",
value: "756c2c21-ef23-4e5a-a63a-ae6aed961d35"
},
{
name: "Ocp-Apim-Subscription-Key",
value: "a2a05eff128943bc89f62b81a63aa368"
},
{
name: "Accept-Charset",
value: "UTF-8"
},
{
name: "Cache-Control",
value: "no-cache"
},
{
name: "Content-Type",
value: "application/query+json"
},
{
name: "Accept",
value: "application/json;odata=nometadata"
},
{
name: "Accept-Encoding",
value: "gzip,deflate"
},
{
name: "Cookie",
value: "x-ms-gateway-slice=008; stsservicecookie=ests; BIGipServerpmc_rest_webservices_http_prod=1242575370.20480.0000"
},
{
name: "User-Agent",
value: "PostmanRuntime/6.2.5"
},
{
name: "x-ms-date",
value: "Wed, 09 Aug 2017 20:10:09 GMT"
},
{
name: "x-ms-version",
value: "2017-02-22"
},
{
name: "MaxDataServiceVersion",
value: "3.0"
},
{
name: "DataServiceVersion",
value: "1.0;NetFx"
},
{
name: "Api-Message-Id",
value: "12427ae7-7704-44cb-b4af-d7e622898b99"
},
{
name: "Api-Identifier",
value: "461f0c19-8df3-4272-9ac7-c64bb776dd56"
},
{
name: "Api-Id",
value: "58987927bceb5204c4e59168"
},
{
name: "Api-Operation-Id",
value: "598b3c72e055cd14fc3abdd1"
},
{
name: "Api-Adapter-Id",
value: "573a1c64bceb520aac127ee5"
},
{
name: "Api-Subscription-Id",
value: "573a1c65bceb52192c140131"
},
{
name: "Api-Policy-Id",
value: "64BC4270-54AC-42DA-835C-E285F35BCA81"
},
{
name: "X-Basic-Username",
value: ""
},
{
name: "x-ms-documentdb-isquery",
value: "True"
},
{
name: "x-ms-documentdb-query-enablecrosspartition",
value: "False"
},
{
name: "x-ms-max-item-count",
value: "1000"
},
{
name: "x-ms-documentdb-partitionkey",
value: "573a1c65bceb52192c140131"
},
{
name: "x-ms-partition-key",
value: "573a1c65bceb52192c140131"
},
{
name: "Authorization",
value: "type=master&ver=1.0&sig=Ke...Q="
},
{
name: "X-Forwarded-For",
value: "75.39.38.67"
}
]

The response I get back is either

{
    "code": "BadRequest",
    "message": "Partition key 573a1c65bceb52192c140131 is invalid.\r\nActivityId: 61836599-fe4b-4232-b55b-2c568eecc767"
}

or

{
    "code": "Unauthorized",
    "message": "The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'post\ndocs\ndbs/WebApi/colls/Logs\nwed, 09 aug 2017 20:35:41 gmt\n\n'\r\nActivityId: 429....2e2"
}

These seems to give me two problems to solve. In the first place how do I go about troubleshooting this partition? It seems to be from my analysis, it is a valid partition, validating by queries in the portal and the headers "x-ms-documentdb-partitionkey" and "x-ms-partition-key". (I have seen both header names in MS documentation, so I am covering my bases with both.)

The "The input authorization token can't serve the request." message suggests to me something variably wrong in my query. I suspect maybe the data value? My policy is little different from one I use for Azure Table Storage REST API and I never have that problem. I am using my read-only primary key taken from Azure portal and stored in API Management's Named Values:

<policies>
    <inbound>
        <base />
        <set-variable name="Content-Type" value="application/query+json" />
        <set-variable name="x-ms-documentdb-isquery" value="True" />
        <set-variable name="x-ms-documentdb-query-enablecrosspartition" value="False" />
        <set-variable name="x-ms-max-item-count" value="1000" />
        <set-variable name="x-ms-version" value="2017-02-22" />
        <set-header name="Content-Type" exists-action="override">
            <value>@((string)context.Variables["Content-Type"])</value>
        </set-header>
        <set-header name="x-ms-documentdb-isquery" exists-action="override">
            <value>@((string)context.Variables["x-ms-documentdb-isquery"])</value>
        </set-header>
        <set-header name="x-ms-documentdb-query-enablecrosspartition" exists-action="override">
            <value>@((string)context.Variables["x-ms-documentdb-query-enablecrosspartition"])</value>
        </set-header>
        <set-header name="x-ms-max-item-count" exists-action="override">
            <value>@((string)context.Variables["x-ms-max-item-count"])</value>
        </set-header>
        <set-header name="x-ms-version" exists-action="override">
            <value>@((string)context.Variables["x-ms-version"])</value>
        </set-header>
        <!-- MS docs may conflict here. Possibly "x-ms-documentdb-partitionkey" req'd and "x-ms-partition-key" not supported -->
        <set-header name="x-ms-documentdb-partitionkey" exists-action="override">
            <value>@(context.Subscription.Id)</value>
        </set-header>
        <set-header name="x-ms-partition-key" exists-action="override">
            <value>@(context.Subscription.Id)</value>
        </set-header>
        <set-variable name="StringToSign" value="@(string.Format("post\ndocs\ndbs/WebApi/colls/Logs\n{0}\n\n", ((string)context.Variables["x-ms-date"]).ToLowerInvariant()))" />
        <set-variable name="cosmosreadonlykey" value="{{CosmosReadOnlyKey}}" />
        <set-variable name="SharedKey" value="@{
        // https://docs.microsoft.com/en-us/rest/api/documentdb/access-control-on-documentdb-resources#constructkeytoken
        System.Security.Cryptography.HMACSHA256 hasher = new System.Security.Cryptography.HMACSHA256(Convert.FromBase64String((string)context.Variables["cosmosreadonlykey"]));
        return Convert.ToBase64String(hasher.ComputeHash(System.Text.Encoding.UTF8.GetBytes((string)context.Variables["StringToSign"])));
}" />
        <set-variable name="Authorization" value="@(string.Format("type=master&ver=1.0&sig={0}", (string)context.Variables["SharedKey"]))" />
        <set-header name="Authorization" exists-action="override">
            <value>@((string)context.Variables["Authorization"])</value>
        </set-header>
        <set-backend-service base-url="https://plexconnectcosmos.documents.azure.com" />
        <rewrite-uri template="/dbs/WebApi/colls/Logs/docs" />
    </inbound>

Some thins I wonder: Could the returned ActivityId help me get any more details, some how? Even without it, is there some logging in Azure I haven't found that would reveal more details.

If anything is obviously wrong here that I am doing, please someone let me know.

Answer 1

I got it running with some minor adjustments.

<policies>
<inbound>
    <base />
    <set-variable name="Content-Type" value="application/query+json" />
    <set-variable name="x-ms-documentdb-isquery" value="True" />
    <set-variable name="x-ms-documentdb-query-enablecrosspartition" value="False" />
    <set-variable name="x-ms-max-item-count" value="1000" />
    <set-variable name="x-ms-version" value="2017-02-22" />
    <set-variable name="x-ms-date" value="@( DateTime.UtcNow.ToString("R") )" />
    <set-header name="Content-Type" exists-action="override">
        <value>@((string)context.Variables["Content-Type"])</value>
    </set-header>
    <set-header name="x-ms-documentdb-isquery" exists-action="override">
        <value>@((string)context.Variables["x-ms-documentdb-isquery"])</value>
    </set-header>
    <set-header name="x-ms-documentdb-query-enablecrosspartition" exists-action="override">
        <value>@((string)context.Variables["x-ms-documentdb-query-enablecrosspartition"])</value>
    </set-header>
    <set-header name="x-ms-max-item-count" exists-action="override">
        <value>@((string)context.Variables["x-ms-max-item-count"])</value>
    </set-header>
    <set-header name="x-ms-version" exists-action="override">
        <value>@((string)context.Variables["x-ms-version"])</value>
    </set-header>
    <set-header name="x-ms-documentdb-partitionkey" exists-action="override">
        <value>@("[\""+context.Subscription.Id+"\"]")</value>
    </set-header>
    <set-header name="x-ms-date" exists-action="override">
        <value>@( (string)context.Variables["x-ms-date"] )</value>
    </set-header>
    <set-variable name="StringToSign" value="@(string.Format("post\ndocs\ndbs/WebApi/colls/Logs\n{0}\n\n", ((string)context.Variables["x-ms-date"]).ToLowerInvariant()))" />
    <set-variable name="cosmosreadonlykey" value="{{CosmosReadOnlyKey}}" />
    <set-variable name="SharedKey" value="@{
    // https://docs.microsoft.com/en-us/rest/api/documentdb/access-control-on-documentdb-resources#constructkeytoken
    System.Security.Cryptography.HMACSHA256 hasher = new System.Security.Cryptography.HMACSHA256(Convert.FromBase64String((string)context.Variables["cosmosreadonlykey"]));
    return Convert.ToBase64String(hasher.ComputeHash(System.Text.Encoding.UTF8.GetBytes((string)context.Variables["StringToSign"])));
}" />
    <set-variable name="Authorization" value="@(string.Format("type=master&ver=1.0&sig={0}", ((string)context.Variables["SharedKey"]).Replace("&","%26").Replace("+","%2B").Replace("=","%3D")))" />
    <set-header name="Authorization" exists-action="override">
        <value>@((string)context.Variables["Authorization"])</value>
    </set-header>
    <set-backend-service base-url="https://mycosmosdb.documents.azure.com" />
    <rewrite-uri template="/dbs/WebApi/colls/Logs/docs" />
</inbound>
</policies>

1. the partition key needs to be formatted as an array
2. date is put into the header and the StringToSign based on the same value
3. did some hacky URL hex encoding - could be improved with proper hex encoding