I have a self made Kubernetes cluster consisting of VMs. My problem is, that the coredns pods are always go in CrashLoopBackOff state, and after a while they go back to Running as nothing happened.. One solution that I found and could not try yet, is changing the default memory limit from 170Mi to something higher. As I'm not an expert in this, I thought this is not a hard thing, but I don't know how to change a running pod's configuration. It may be impossible, but there must be a way to recreate them with new configuration. I tried with kubectl patch, and looked up rolling-update too, but I just can't figure it out. How can I change the limit?
Here is the relevant part of the pod's data:
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/podIP: 176.16.0.12/32
creationTimestamp: 2018-11-18T10:29:53Z
generateName: coredns-78fcdf6894-
labels:
k8s-app: kube-dns
pod-template-hash: "3497892450"
name: coredns-78fcdf6894-gnlqw
namespace: kube-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: coredns-78fcdf6894
uid: e3349719-eb1c-11e8-9000-080027bbdf83
resourceVersion: "73564"
selfLink: /api/v1/namespaces/kube-system/pods/coredns-78fcdf6894-gnlqw
uid: e34930db-eb1c-11e8-9000-080027bbdf83
spec:
containers:
- args:
- -conf
- /etc/coredns/Corefile
image: k8s.gcr.io/coredns:1.1.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 5
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: coredns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
EDIT: It turned out, that in Ubuntu the Network Manager's dnsmasq drives the Corends pods crazy, so in /etc/NetworkManager/NetworkManager.conf I commented out the dnsmasq line, reboot and everything is okay.
You can change via Azure CLI - Specify the --max-pods argument when you deploy a cluster with the az aks create command. The maximum value is 250. You can't change the maximum number of pods per node when you deploy a cluster with the Azure portal.
If a container attempts to exceed the specified limit, the system will throttle the container.
By default, containers run with unbounded compute resources on a Kubernetes cluster. Using Kubernetes resource quotas, administrators (also termed cluster operators) can restrict consumption and creation of cluster resources (such as CPU time, memory, and persistent storage) within a specified namespace.
When you specify the resource request for containers in a Pod, the kube-scheduler uses this information to decide which node to place the Pod on. When you specify a resource limit for a container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set.
Setting resource limits on your Kubernetes pods prevents an errant container from impacting other workloads. Kubernetes lets you cap resources, including CPU and memory consumption. Pods can be terminated when their limits are exceeded, maintaining the overall stability of the cluster. Sorry, the video player failed to load. (Error Code: 100013)
Both containers are defined with a request for 0.25 CPU and 64MiB (2 26 bytes) of memory. Each container has a limit of 0.5 CPU and 128MiB of memory. You can say the Pod has a request of 0.5 CPU and 128 MiB of memory, and a limit of 1 CPU and 256MiB of memory. When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on.
Requests and limits are the mechanisms Kubernetes uses to control resources such as CPU and memory. Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource. Limits, on the other hand, make sure a container never goes above a certain value.
You must edit coredns pod's template in coredns deployment definition:
kubectl edit deployment -n kube-system coredns
Once your default editor is opened with coredns deployment, in the templateSpec you will find part which is responsible for setting memory and cpu limits.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With