Error: selector does not match template labels

Question

My task is to add a label named "app" to all deployments, daemonsets, and cronjobs so that it's easier to query our apps across the stack in our monitoring tools. This way, we can build dashboards that use a single selector, namely app.

To avoid downtime I've decided to resolve this issue in the following steps:

1. Add labels to dev, test & stage environments.
2. Add labels to prod env's.
3. Deploy (1)
4. Deploy (2)
5. Delete old labels & update the services of dev to use the new labels. Then test & deploy. (currently on this step)
6. Repeat (5) for stage.
7. Repeat (5) for prod.

When using $ kubectl apply to update the resources I've added the "app" label to/replaced "service" label with "app" labels to, I run into the following error:

Error from server (Invalid): error when applying patch: {longAssPatchWhichIWon'tIncludeButYaGetThePoint} to: &{0xc421b02f00 0xc420803650 default provisioning manifests/prod/provisioning-deployment.yaml 0xc 42000c6f8 3942200 false} for: "manifests/prod/provisioning-deployment.yaml": Deployment.apps "provisioning" is invalid: s pec.template.metadata.labels: Invalid value: map[string]string{"app":"provisioning", "component" :"marketplace"}: selector does not match template labels

I need some insights on why it's throwing this error.

Answer 1

It seems you are in trouble. Check this section: Label selector updates

Note: In API version apps/v1, a Deployment’s label selector is immutable after it gets created.

So, this line say you can not update selector once deployment is created. Selector can not be changed for any API version except apps/v1beta1 and extension/v1beta1. Ref: TestDeploymentSelectorImmutability.

One possible workaround might be to keep the old labels and adding new labels along with old ones. This way, you don't have to update selector. Deployment will select pods using old labels but your dashboard can select using new labels. This might not meet your requirement but I don't see any better way.