upstream connect error or disconnect/reset before headers. reset reason: connection failure. Spring Boot and java 11

Question

I'm having a problem migrating my pure Kubernetes app to an Istio managed. I'm using Google Cloud Platform (GCP), Istio 1.4, Google Kubernetes Engine (GKE), Spring Boot and JAVA 11.

I had the containers running in a pure GKE environment without a problem. Now I started the migration of my Kubernetes cluster to use Istio. Since then I'm getting the following message when I try to access the exposed service.

upstream connect error or disconnect/reset before headers. reset reason: connection failure

This error message looks like a really generic. I found a lot of different problems, with the same error message, but no one was related to my problem.

Bellow the version of the Istio:

client version: 1.4.10
control plane version: 1.4.10-gke.5
data plane version: 1.4.10-gke.5 (2 proxies)

Bellow my yaml files:

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    account: tree-guest
  name: tree-guest-service-account
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: tree-guest
    service: tree-guest
  name: tree-guest
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: tree-guest
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: tree-guest
    version: v1
  name: tree-guest-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tree-guest
      version: v1
  template:
    metadata:
      labels:
        app: tree-guestaz
        version: v1
    spec:
      containers:
      - image: registry.hub.docker.com/victorsens/tree-quest:circle_ci_build_00923285-3c44-4955-8de1-ed578e23c5cf
        imagePullPolicy: IfNotPresent
        name: tree-guest
        ports:
        - containerPort: 8080
      serviceAccount: tree-guest-service-account
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tree-guest-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tree-guest-virtual-service
spec:
  hosts:
    - "*"
  gateways:
    - tree-guest-gateway
  http:
    - match:
        - uri:
            prefix: /v1
      route:
        - destination:
            host: tree-guest
            port:
              number: 8080

To apply the yaml file I used the following argument:

kubectl apply -f <(istioctl kube-inject -f ./tree-guest.yaml)

Below the result of the Istio proxy argument, after deploying the application:

istio-ingressgateway-6674cc989b-vwzqg.istio-system SYNCED SYNCED SYNCED SYNCED 
istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5 tree-guest-v1-774bf84ddd-jkhsh.default SYNCED SYNCED SYNCED SYNCED istio-pilot-ff4489db8-2hx5f 1.4.10-gke.5

If someone have a tip about what is going wrong, please let me know. I'm stuck in this problem for a couple of days.

Thanks.

Answer 1

As @Victor mentioned the problem here was the wrong yaml file.

I solve it. In my case the yaml file was wrong. I reviewed it and the problem now is solved. Thank you guys., – Victor

If you're looking for yaml samples I would suggest to take a look at istio github samples.

---

As 503 upstream connect error or disconnect/reset before headers. reset reason: connection failure occurs very often I set up little troubleshooting answer, there are another questions with 503 error which I encountered for several months with answers, useful informations from istio documentation and things I would check.

Examples with 503 error:

• Istio 503:s between (Public) Gateway and Service
• IstIO egress gateway gives HTTP 503 error
• Istio Ingress Gateway with TLS termination returning 503 service unavailable
• how to terminate ssl at ingress-gateway in istio?
• Accessing service using istio ingress gives 503 error when mTLS is enabled

Common cause of 503 errors from istio documentation:

• https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes
• https://istio.io/docs/ops/common-problems/network-issues/#503-errors-after-setting-destination-rule
• https://istio.io/latest/docs/concepts/traffic-management/#working-with-your-applications

Few things I would check first:

• Check services ports name, Istio can route correctly the traffic if it knows the protocol. It should be <protocol>[-<suffix>] as mentioned in istio documentation.
• Check mTLS, if there are any problems caused by mTLS, usually those problems would result in error 503.
• Check if istio works, I would recommend to apply bookinfo application example and check if it works as expected.
• Check if your namespace is injected with kubectl get namespace -L istio-injection
• If the VirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.