Phoenix channel's socket keeps getting closed in distributed cluster environment

Question

I followed this series of articles to create a deployment at Google Cloud. Everything is working as expected, except phoenix channels.

No errors on the backend side. On Javascript frontend I am getting first channel ERROR and then the socket CLOSES on frontend while using channels. And this keeps repeating endlessly in an interval of 10–20 secs.

CHANNEL ERROR!
SOCKET CLOSE!
CHANNEL ERROR!
SOCKET CLOSE!

From this code:

socket.connect()
socket.onError( e => console.log("SOCKET ERROR", e) )
socket.onClose( e => console.log("SOCKET CLOSE", e))
channel = socket.channel("note:" + noteId, {})
channel.onError( e => console.log("CHANNEL ERROR!", e) )
channel.onClose( e => console.log("CHANNEL CLOSED!", e) )

I need help to debug this and figure out where this problem is originating from. Please let me know if any piece of code is needed and I will update the question with that code. Its been a week now. :(

Thanks a lot!

(No problem when run locally)

UPDATE: The only difference I am seeing is that on local server, phoenix.js is continuously sending heartbeat but this is not happening on server.

UPDATE:

---- my-ingress.yaml ----
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    # ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.global-static-ip-name: my-ingress-ip
    kubernetes.io/tls-acme: "true"
spec:
  rules:
  - host: apiv2.example.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: my-service-nodeport
          servicePort: 80
      - path: /.well-known/acme-challenge
        backend:
          serviceName: kube-lego-gce
          servicePort: 8080
  tls:
  - secretName: my-secret-tls-7
    hosts:
    - apiv2.example.com

This is ingress file I am using and also using kube-lego with it.

UPDATE: I implemented this code (I checked frames in dev tools, they were being sent). But still... its giving SOCKER ERROR. May be issue is not with sending hearbeats.

const HEARTBEAT_INTERVAL_MS = 5000
socket.onOpen(() => {
      clearInterval(this.heartbeatTimer);
      this.heartbeatTimer = setInterval(() => { 
        return socket.push({ 
          topic: "phoenix", 
          event: "heartbeat", 
          payload: {}, 
          ref: socket.makeRef(), 
        }); 
      }, 
        HEARTBEAT_INTERVAL_MS
      );
    })

Answer 1

You may need to increase the response timeout for your backend service on GCP.

By default, the timeout was set at 30sec and was causing the same problem for me.