Programmatic access from a service account to a Google IAP protected resource denied with invalid signature error

Question

I recently enabled IAP in GKE cluster.

• Cluster Version: 1.15.11-gke.11

I followed the instructions here: https://cloud.google.com/iap/docs/enabling-kubernetes-howto

Service config is as follows:

---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: foo-bc-iap
  namespace: foo-test
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: iap-client-secret
---
apiVersion: v1
kind: Service
metadata:
  name: foo-internal-service
  namespace: foo-test
  annotations:
    cloud.google.com/backend-config: '{"ports":{"80":"foo-bc-iap"}}'
spec:
  type: NodePort # To create Ingress using the service.
  selector:
    app: foo-test
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8081

The credential I used was OAuth 2.0 Client ID (Type: Web Application).

After making sure the IAP-protected API endpoint works differently when I activate IAP on the Kubernetes service, I wrote the following test program to make sure the endpoint is accessible from the service account given in the JSON file 'account.json'.

In writing this sample application, I consulted this doc: https://cloud.google.com/iap/docs/authentication-howto#iap_make_request-go

• google.golang.org/api v0.23.0
• go 1.12

func (m *myApp) testAuthz(ctx *cli.Context) error {
    audience := "<The client ID of the credential mentioned above>"
    serviceAccountOption := idtoken.WithCredentialsFile("account.json")

    client, err := idtoken.NewClient(ctx.Context, audience, serviceAccountOption)
    if err != nil {
        return fmt.Errorf("idtoken.NewClient: %v", err)
    }

    requestBody := `{
        <some JSON payload>
    }`

    request, err := http.NewRequest("POST", "https://my.iap.protected/endpoint",
        bytes.NewBuffer([]byte(requestBody)))
    if err != nil {
        return fmt.Errorf("http.NewRequest: %v", err)
    }

    request.Header.Add("Content-Type", "application/json")

    response, err := client.Do(request)
    if err != nil {
        return fmt.Errorf("client.Do: %v", err)
    }
    defer response.Body.Close()

    fmt.Printf("request header = %#v\n", response.Request.Header)
    fmt.Printf("response header = %#v\n", response.Header)

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        return fmt.Errorf("ioutil.ReadAll: %v", err)
    }

    fmt.Printf("%d: %s\n", response.StatusCode, string(body))

    return nil
}

But when I run this, I could only see the following response.

request header = http.Header{"Authorization":[]string{"Bearer <jwt token>"}, "Content-Type":[]string{"application/json"}, "X-Cloud-Trace-Context":[]string{"c855757f20d155da1140fad1508ae3e5/17413578722158830486;o=0"}}

response header = http.Header{"Alt-Svc":[]string{"clear"}, "Content-Length":[]string{"49"}, "Content-Type":[]string{"text/html; charset=UTF-8"}, "Date":[]string{"Wed, 06 May 2020 22:17:43 GMT"}, "X-Goog-Iap-Generated-Response":[]string{"true"}}

401: Invalid IAP credentials: JWT signature is invalid

As you can see here, the access was denied.

So I thought that the signature used to sign the JWT token in the header might be wrong.

But I made sure the following using jwt.io:

• The JWT token used in the header is signed by the private key of the caller's service account
• The JWT token used in the header can be verified by the public key of the caller's service account
• The JWT token was signed using RS256 algorithm

And I also looked into the token:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "<the service account's private key ID>"
}

{
  "iss": "<email address of the service account>",
  "aud": "",
  "exp": 1588806087,
  "iat": 1588802487,
  "sub": "<email address of the service acocunt>"
}

Nothing quite odd.

So I'm not sure what's going on here. If I disable IAP, the endpoint return the right response.

Can anyone give me some hint about what I am doing wrong?

Answer 1

I tried your code, and I found that it doesn't work with google.golang.org/api v0.23.0, but it does work with google.golang.org/api v0.24.0 (latest one at the time of writing).

It is indeed a bug, the release notes mention the following:

When provided, use the TokenSource from options for NewTransport. This fixes a bug in idtoken.NewClient where the wrong TokenSource was being used for authentication.

Interestingly, 0.23.0 is sending a token signed with the service account's private key with the following claims:

{
  "iss":"[email protected]",
  "aud":"",
  "exp":1589596554,
  "iat":1589592954,
  "sub":"[email protected]"
}

and 0.24.0 sends a token signed with google's private key. (internally the previous token is exchanged for a google-signed token)

{
    "aud":"xxxxxxx-xxxxxxxxxxxxxxxxxx.apps.googleusercontent.com",
    "azp":"[email protected]",
    "email":"[email protected]",
    "email_verified":true,
    "exp":1589596508,
    "iat":1589592908,
    "iss":"https://accounts.google.com",
    "sub":"11524xxxxxxxxxxxxxxxx"
}