Error: UPGRADE FAILED: failed to replace object: Service "api" is invalid: spec.clusterIP: Invalid value: "": field is immutable

Question

When doing helm upgrade ... --force I'm getting this below error

Error: UPGRADE FAILED: failed to replace object: Service "api" is invalid: spec.clusterIP: Invalid value: "": field is immutable

And This is how my service file looks like: (Not passing clusterIP anywhere )

apiVersion: v1
kind: Service
metadata:
  name: {{ .Chart.Name }}
  namespace: {{ .Release.Namespace }}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
  labels:
    app: {{ .Chart.Name }}-service
    kubernetes.io/name: {{ .Chart.Name | quote }}
    dns: route53
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
spec:
  selector:
    app: {{ .Chart.Name }}
  type: LoadBalancer
  ports:
  - port: 443
    name: https
    targetPort: http-port
    protocol: TCP

Helm Version: 3.0.1

Kubectl Version: 1.13.1 [Tried with the 1.17.1 as well]

Server: 1.14

Note: Previously I was using some old version (of server, kubectl, helm) at that time I did not face this kind of issue. I can see lots of similar issues in GitHub regarding this, but unable to find any working solution for me.

few of the similar issues:

https://github.com/kubernetes/kubernetes/issues/25241

https://github.com/helm/charts/pull/13646 [For Nginx chart]

Answer 1

I've made some tests with Helm and got the same issue when trying to change the Service type from NodePort/ClusterIP to LoadBalancer.

This is how I've reproduced your issue:

Kubernetes 1.15.3 (GKE) Helm 3.1.1

Helm chart used for test: stable/nginx-ingress

How I reproduced:

1. Get and decompress the file:

helm fetch stable/nginx-ingress  
tar xzvf nginx-ingress-1.33.0.tgz  

2. Modify service type from type: LoadBalancer to type: NodePort in the values.yaml file (line 271):

sed -i '271s/LoadBalancer/NodePort/' values.yaml

3. Install the chart:

helm install nginx-ingress ./

4. Check service type, must be NodePort:

kubectl get svc -l app=nginx-ingress,component=controller

NAME                       TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
nginx-ingress-controller   NodePort   10.0.3.137   <none>        80:30117/TCP,443:30003/TCP   1m

5. Now modify the Service type again to LoadBalancer in the values.yaml:

sed -i '271s/NodePort/LoadBalancer/' values.yaml

6. Finally, try to upgrade the chart using --force flag:

helm upgrade nginx-ingress ./ --force

And then:

Error: UPGRADE FAILED: failed to replace object: Service "nginx-ingress-controller" is invalid: spec.clusterIP: Invalid value: "": field is immutable

Explanation

Digging around I found this in HELM source code:

// if --force is applied, attempt to replace the existing resource with the new object.
    if force {
        obj, err = helper.Replace(target.Namespace, target.Name, true, target.Object)
        if err != nil {
            return errors.Wrap(err, "failed to replace object")
        }
        c.Log("Replaced %q with kind %s for kind %s\n", target.Name, currentObj.GetObjectKind().GroupVersionKind().Kind, kind)
    } else {
        // send patch to server
        obj, err = helper.Patch(target.Namespace, target.Name, patchType, patch, nil)
        if err != nil {
            return errors.Wrapf(err, "cannot patch %q with kind %s", target.Name, kind)
        }
    }

Analyzing the code above Helm will use similar to kubectl replace api request (instead of kubectl replace --force as we could expect)... when the helm --force flag is set.

If not, then Helm will use kubectl patch api request to make the upgrade.

Let's check if it make sense:

PoC using kubectl

1. Create a simple service as NodePort:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
 labels:
   app: test-svc
 name: test-svc
spec:
 selector:
   app: test-app
 ports:
 - port: 80
   protocol: TCP
   targetPort: 80
 type: NodePort
EOF

Make the service was created:

kubectl get svc -l app=test-svc

NAME       TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
test-svc   NodePort   10.0.7.37    <none>        80:31523/TCP   25

Now lets try to use kubectl replace to upgrade the service to LoadBalancer, like helm upgrade --force:

kubectl replace -f - <<EOF
apiVersion: v1
kind: Service
metadata:
 labels:
   app: test-svc
 name: test-svc
spec:
 selector:
   app: test-app
 ports:
 - port: 80
   protocol: TCP
   targetPort: 80
 type: LoadBalancer
EOF

This shows the error:

The Service "test-svc" is invalid: spec.clusterIP: Invalid value: "": field is immutable

Now, lets use kubectl patch to change the NodePort to LoadBalancer, simulating the helm upgrade command without --force flag:

Here is the kubectl patch documentation, if want to see how to use.

kubectl patch svc test-svc -p '{"spec":{"type":"LoadBalancer"}}'

Then you see: service/test-svc patched

Workaround

You should to use helm upgrade without --force, it will work.

If you really need to use --force to recreate some resources, like pods to get the latest configMap update, for example, then I suggest you first manually change the service specs before Helm upgrade.

If you are trying to change the service type you could do it exporting the service yaml, changing the type and apply it again (because I experienced this behavior only when I tried to apply the same template from the first time):

kubectl get svc test-svc -o yaml | sed 's/NodePort/LoadBalancer/g' | kubectl replace --force -f -

The Output:

service "test-svc" deleted
service/test-svc replaced

Now, if you try to use helm upgrade --force and doesn't have any change to do in the service, it will work and will recreate your pods and others resources.

I hope that helps you!