GCloud kubernetes cluster with 1 Insufficient cpu error

Question

I created a Kubernetes cluster on Google Cloud using:

gcloud container clusters create my-app-cluster --num-nodes=1

Then I deployed my 3 apps (backend, frontend and a scraper) and created a load balancer. I used the following configuration file:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: my-app-deployment
    labels:
        app: my-app
spec:
    replicas: 1
    selector:
        matchLabels:
            app: my-app
    template:
        metadata:
            labels:
                app: my-app
        spec:
            containers:
              - name: my-app-server
                image: gcr.io/my-app/server
                ports:
                  - containerPort: 8009
                envFrom:
                  - secretRef:
                        name: my-app-production-secrets
              - name: my-app-scraper
                image: gcr.io/my-app/scraper
                ports:
                  - containerPort: 8109
                envFrom:
                  - secretRef:
                        name: my-app-production-secrets
              - name: my-app-frontend
                image: gcr.io/my-app/frontend
                ports:
                  - containerPort: 80
                envFrom:
                  - secretRef:
                        name: my-app-production-secrets

---

apiVersion: v1
kind: Service
metadata:
    name: my-app-lb-service
spec:
    type: LoadBalancer
    selector:
        app: my-app
    ports:
      - name: my-app-server-port
        protocol: TCP
        port: 8009
        targetPort: 8009
      - name: my-app-scraper-port
        protocol: TCP
        port: 8109
        targetPort: 8109
      - name: my-app-frontend-port
        protocol: TCP
        port: 80
        targetPort: 80

When typing kubectl get pods I get:

NAME                                   READY     STATUS    RESTARTS   AGE
my-app-deployment-6b49c9b5c4-5zxw2   0/3       Pending   0          12h

When investigation i Google Cloud I see "Unschedulable" state with "insufficient cpu" error on pod:

When going to Nodes section under my cluster in the Clusters page, I see 681 mCPU requested and 940 mCPU allocated:

What is wrong? Why my pod doesn't start?

Answer 1

Every container has a default CPU request (in GKE I’ve noticed it’s 0.1 CPU or 100m). Assuming these defaults you have three containers in that pod so you’re requesting another 0.3 CPU.

The node has 0.68 CPU (680m) requested by other workloads and a total limit (allocatable) on that node of 0.94 CPU (940m).

If you want to see what workloads are reserving that 0.68 CPU, you need to inspect the pods on the node. In the page on GKE where you see the resource allocations and limits per node, if you click the node it will take you to a page that provides this information.
In my case I can see 2 pods of kube-dns taking 0.26 CPU each, amongst others. These are system pods that are needed to operate the cluster correctly. What you see will also depend on what add-on services you have selected, for example: HTTP Load Balancing (Ingress), Kubernetes Dashboard and so on.

Your pod would take CPU to 0.98 CPU for the node which is more than the 0.94 limit, which is why your pod cannot start.

Note that the scheduling is based on the amount of CPU requested for each workload, not how much it actually uses, or the limit.

Your options:

1. Turn off any add-on service which is taking CPU resource that you don't need.
2. Add more CPU resource to your cluster. To do that you will either need to change your node pool to use VMs with more CPU, or increase the number of nodes in your existing pool. You can do this in GKE console or via the gcloud command line.
3. Make explicit requests in your containers for less CPU that will override the defaults.

apiVersion: apps/v1
kind: Deployment
...
        spec:
            containers:
              - name: my-app-server
                image: gcr.io/my-app/server
                ...
                resources:
                  requests:
                     cpu: "50m"
              - name: my-app-scraper
                image: gcr.io/my-app/scraper
                ...
                resources:
                  requests:
                     cpu: "50m"
              - name: my-app-frontend
                image: gcr.io/my-app/frontend
                ...
                resources:
                  requests:
                     cpu: "50m"