Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to change running pods limits in Kubernetes?

I have a self made Kubernetes cluster consisting of VMs. My problem is, that the coredns pods are always go in CrashLoopBackOff state, and after a while they go back to Running as nothing happened.. One solution that I found and could not try yet, is changing the default memory limit from 170Mi to something higher. As I'm not an expert in this, I thought this is not a hard thing, but I don't know how to change a running pod's configuration. It may be impossible, but there must be a way to recreate them with new configuration. I tried with kubectl patch, and looked up rolling-update too, but I just can't figure it out. How can I change the limit?

Here is the relevant part of the pod's data:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 176.16.0.12/32
  creationTimestamp: 2018-11-18T10:29:53Z
  generateName: coredns-78fcdf6894-
  labels:
    k8s-app: kube-dns
    pod-template-hash: "3497892450"
  name: coredns-78fcdf6894-gnlqw
  namespace: kube-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: coredns-78fcdf6894
    uid: e3349719-eb1c-11e8-9000-080027bbdf83
  resourceVersion: "73564"
  selfLink: /api/v1/namespaces/kube-system/pods/coredns-78fcdf6894-gnlqw
  uid: e34930db-eb1c-11e8-9000-080027bbdf83
spec:
  containers:
  - args:
    - -conf
    - /etc/coredns/Corefile
  image: k8s.gcr.io/coredns:1.1.3
  imagePullPolicy: IfNotPresent
  livenessProbe:
    failureThreshold: 5
    httpGet:
      path: /health
      port: 8080
      scheme: HTTP
    initialDelaySeconds: 60
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
  name: coredns
  ports:
  - containerPort: 53
    name: dns
    protocol: UDP
  - containerPort: 53
    name: dns-tcp
    protocol: TCP
  - containerPort: 9153
    name: metrics
    protocol: TCP
  resources:
    limits:
      memory: 170Mi
    requests:
      cpu: 100m
      memory: 70Mi

EDIT: It turned out, that in Ubuntu the Network Manager's dnsmasq drives the Corends pods crazy, so in /etc/NetworkManager/NetworkManager.conf I commented out the dnsmasq line, reboot and everything is okay.

like image 656
B.G. Avatar asked Nov 23 '18 14:11

B.G.


People also ask

How do I change my max pods in Kubernetes?

You can change via Azure CLI - Specify the --max-pods argument when you deploy a cluster with the az aks create command. The maximum value is 250. You can't change the maximum number of pods per node when you deploy a cluster with the Azure portal.

What happens if pod exceeds CPU limit?

If a container attempts to exceed the specified limit, the system will throttle the container.

What is limit range in Kubernetes?

By default, containers run with unbounded compute resources on a Kubernetes cluster. Using Kubernetes resource quotas, administrators (also termed cluster operators) can restrict consumption and creation of cluster resources (such as CPU time, memory, and persistent storage) within a specified namespace.

How does Kubernetes decide which node to place a pod on?

When you specify the resource request for containers in a Pod, the kube-scheduler uses this information to decide which node to place the Pod on. When you specify a resource limit for a container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set.

Why should I set resource limits on my Kubernetes pods?

Setting resource limits on your Kubernetes pods prevents an errant container from impacting other workloads. Kubernetes lets you cap resources, including CPU and memory consumption. Pods can be terminated when their limits are exceeded, maintaining the overall stability of the cluster. Sorry, the video player failed to load. (Error Code: 100013)

How many CPU and memory does a Kubernetes pod have?

Both containers are defined with a request for 0.25 CPU and 64MiB (2 26 bytes) of memory. Each container has a limit of 0.5 CPU and 128MiB of memory. You can say the Pod has a request of 0.5 CPU and 128 MiB of memory, and a limit of 1 CPU and 256MiB of memory. When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on.

What is the difference between requests and limits in Kubernetes?

Requests and limits are the mechanisms Kubernetes uses to control resources such as CPU and memory. Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource. Limits, on the other hand, make sure a container never goes above a certain value.


1 Answers

You must edit coredns pod's template in coredns deployment definition:

kubectl edit deployment -n kube-system coredns

Once your default editor is opened with coredns deployment, in the templateSpec you will find part which is responsible for setting memory and cpu limits.

like image 194
Ottovsky Avatar answered Oct 04 '22 19:10

Ottovsky