How to perform a rolling update for two dependent deployments where one of them has a longer grace period?

Question

I have a Kubernetes cluster with the following components(Nginx-ingress).

When I deploy a new version of the Tomcat, old ones will stick around for a while(for application reasons) maybe like 10 minutes then they will die. For a short period of time, some users will connect to the old pods and some will connect to new ones. Users who connect to V2 of tomcat need to get the new static assets meaning they need to connect to Apache v2. The problem is that even if I keep both versions of the apache around for the same amount of the time, how am I going to make sure users with the new version of tomcat connect to the new version of apache and users with the old version of tomcat connect to the old version of apache? I want apache pods to be stateless.

Answer 1

I think you should try the blue green deployment model.

A blue/green deployment is a change management strategy for releasing software code. Blue/green deployments, which may also be referred to as A/B deployments require two identical hardware environments that are configured exactly the same way. While one environment is active and serving end users, the other environment remains idle.

Container technology offers a stand-alone environment to run the desired service, which makes it super easy to create identical environments as required in the blue/green deployment. The loosely coupled Services - ReplicaSets, and the label/selector-based service routing in Kubernetes make it easy to switch between different backend environments. With these techniques, the blue/green deployments in Kubernetes can be done as follows:

• Before the deployment, the infrastructure is prepared like so:
  • Prepare the blue deployment and green deployment with TOMCAT_VERSION=7 and TARGET_ROLE set to blue or green respectively.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tomcat-deployment-${TARGET_ROLE}
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: tomcat
        role: ${TARGET_ROLE}
    spec:
      containers:
      - name: tomcat-container
        image: tomcat:${TOMCAT_VERSION}
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /
            port: 8080

Prepare the public service endpoint, which initially routes to one of the backend environments, say TARGET_ROLE=blue.

kind: Service
apiVersion: v1
metadata:
  name: tomcat-service
  labels:
    app: tomcat
    role: ${TARGET_ROLE}
    env: prod
spec:
  type: LoadBalancer
  selector:
    app: tomcat
    role: ${TARGET_ROLE}
  ports:
    - port: 80
      targetPort: 8080

Optionally, prepare a test endpoint so that we can visit the backend environments for testing. They are similar to the public service endpoint, but they are intended to be accessed internally by the dev/ops team only.

kind: Service
apiVersion: v1
metadata:
  name: tomcat-test-${TARGET_ROLE}
  labels:
    app: tomcat
    role: test-${TARGET_ROLE}
spec:
  type: LoadBalancer
  selector:
    app: tomcat
    role: ${TARGET_ROLE}
  ports:
    - port: 80
      targetPort: 8080

• Update the application in the inactive environment, say green environment. Set TARGET_ROLE=green and TOMCAT_VERSION=8 in the deployment config to update the green environment.
• Test the deployment via the tomcat-test-green test endpoint to ensure the green environment is ready to serve client traffic.
• Switch the frontend Service routing to the green environment by updating the Service config with TARGET_ROLE=green.
• Run additional tests on the public endpoint to ensure it is working properly.
• Now the blue environment is idle and we can:
  • leave it with the old application so that we can roll back if there's issue with the new application
  • update it to make it a hot backup of the active environment
  • reduce its replica count to save the occupied resources

As compared to Rolling Update, the blue/green up* The public service is either routed to the old applications, or new applications, but never both at the same time.

• The time it takes for the new pods to be ready does not affect the public service quality, as the traffic will only be routed to the new pods when all of them are tested to be ready.
• We can do comprehensive tests on the new environment before it serves any public traffic. Just keep in mind this is in production, and the tests should not pollute live application data.

References