GKE Instance Metadata pod logging "Unable to sync sandbox" millions of times

Question

I have enabled Workload Identity on my GKE cluster(s) and noticed that the gke-instance-metadata pods started logging an insane amount, with two log lines coming up in particular:

1. "Unable to sync sandbox "05341100f9a2739c19a9c2772a17d4384005c934882b118f183a83c4cc1d558c": setup not complete: podUID="cbef7961-c864-4f21-8861-45f99b2eee90", ip="""
2. ""/computeMetadata/v1/instance/attributes/cluster-name" HTTP/200"

I now have a total of over 348 million log entries over 7 days which is affecting my billing.

Anyone have an idea why this is happening, and how to stop it?

Answer 1

I spun up a fresh cluster with workload identity enabled.

I used the following log explorer query:

resource.type="k8s_container"
jsonPayload.message:"Unable to sync sandbox" 
resource.labels.container_name="gke-metadata-server"

and immediately saw the same log message.

It's my understanding that that's a benign message / normal log message that you can / should ignore.

It's triggered by creating pods I ran kubectl run nginx --image=nginx, and immediately saw the message. (I think it's also triggered by pods being deleted as I saw it again when cluster autoscaler scaled down some nodes.)