How can I avoid dropped connections on AWS EKS?

Question

I have a simple single-pod postgresql deployment running on AWS EKS (code here). I have exposed the pod using a load balancer.

kubectl get services/postgres-lb -o yaml

This yields the following:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "false"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  creationTimestamp: 2019-04-23T02:36:54Z
  labels:
    app: postgres
  name: postgres-lb
  namespace: default
  resourceVersion: "1522157"
  selfLink: /api/v1/namespaces/default/services/postgres-lb
  uid: <HASHREMOVED?
spec:
  clusterIP: 10.100.94.170
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 32331
    port: 5434
    protocol: TCP
    targetPort: 5432
  selector:
    app: postgres
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: ...aaadz-example.elb.us-east-1.amazonaws.com

This works and I can access the pod as expected. However, the connection to postgresql seems to drop about every minute or so if not active. I am pretty sure that at least some AWS load balancers behave this way to "drain connections"; hence the annotation above to NOT drain connections. However, I still see the same behavior of dropping connections if idle.

What is the best practice on AWS EKS for hosting a database, for example, and then exposing its single port to the internet? Web searches have turned up many variations, but all seem either overly complicated or not directly applicable. I have used GCE and found it to be much more straightforward with respect to network and exposing ports, so I feel like I am missing something obvious on AWS.

Answer 1

Try setting service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout annotation to some larger value (by default it is 60 seconds in AWS).