Menu
I'm not sure why I'm getting an error No nodes are available that match all of the following predicates:: Insufficient cpu (1).
I don't recall setting any CPU limits. Unless this is some default?
The output of kubectl describe pod wordpress:
Name: wordpress-114465096-bn4rv
Namespace: default
Node: /
Labels: app=wordpress
pod-template-hash=114465096
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"wordpress-114465096","uid":"fff460df-7c4c-11e7-b3fd-42010a840026...
kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container wordpress; cpu request for container cloudsql-proxy; cpu request for container nginx
Status: Pending
IP:
Controllers: ReplicaSet/wordpress-114465096
Containers:
wordpress:
Image: wordpress:latest
Port:
Requests:
cpu: 100m
Environment:
WORDPRESS_HOST: localhost
WORDPRESS_DB_USERNAME: <set to the key 'username' in secret 'cloudsql-db-credentials'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
/var/www/html from wordpress-persistent-storage (rw)
cloudsql-proxy:
Image: gcr.io/cloudsql-docker/gce-proxy:1.09
Port:
Command:
/cloud_sql_proxy
--dir=/cloudsql
-instances=inspiring-tower-99712:europe-west1:wordpressdb=tcp:3306
-credential_file=/secrets/cloudsql/credentials.json
Requests:
cpu: 100m
Environment: <none>
Mounts:
/cloudsql from cloudsql (rw)
/etc/ssl/certs from ssl-certs (rw)
/secrets/cloudsql from cloudsql-instance-credentials (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
nginx:
Image: nginx:latest
Port: 80/TCP
Requests:
cpu: 100m
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
wordpress-persistent-storage:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: wordpress-disk
FSType: ext4
Partition: 0
ReadOnly: false
cloudsql-instance-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: cloudsql-instance-credentials
Optional: false
ssl-certs:
Type: HostPath (bare host directory volume)
Path: /etc/ssl/certs
cloudsql:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-ql6k8:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ql6k8
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 22s 265 default-scheduler Warning FailedScheduling No nodes are available that match all of the following predicates:: Insufficient cpu (1).
Output of kubectl describe node gke-wordpress-default-pool-91c14317-jdlj (the single node in the cluster):
Name: gke-wordpress-default-pool-91c14317-jdlj
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-1
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=default-pool
failure-domain.beta.kubernetes.io/region=europe-west1
failure-domain.beta.kubernetes.io/zone=europe-west1-b
kubernetes.io/hostname=gke-wordpress-default-pool-91c14317-jdlj
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 04 Aug 2017 17:44:08 +0100
Phase:
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Fri, 04 Aug 2017 17:44:35 +0100 Fri, 04 Aug 2017 17:44:35 +0100 RouteCreated RouteController created a route
OutOfDisk False Tue, 08 Aug 2017 21:04:47 +0100 Fri, 04 Aug 2017 17:44:08 +0100 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Tue, 08 Aug 2017 21:04:47 +0100 Fri, 04 Aug 2017 17:44:08 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 08 Aug 2017 21:04:47 +0100 Fri, 04 Aug 2017 17:44:08 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
Ready True Tue, 08 Aug 2017 21:04:47 +0100 Fri, 04 Aug 2017 17:44:39 +0100 KubeletReady kubelet is posting ready status. AppArmor enabled
KernelDeadlock False Tue, 08 Aug 2017 21:03:56 +0100 Fri, 04 Aug 2017 17:43:19 +0100 KernelHasNoDeadlock kernel has no deadlock
Addresses: 10.132.0.3,35.195.163.26,gke-wordpress-default-pool-91c14317-jdlj
Capacity:
cpu: 1
memory: 3794520Ki
pods: 110
Allocatable:
cpu: 1
memory: 3794520Ki
pods: 110
System Info:
Machine ID: 2643dae58dd36381dc5e8ebe124272bc
System UUID: 2643DAE5-8DD3-6381-DC5E-8EBE124272BC
Boot ID: 37002900-44ab-45b1-bbca-04d2b5866683
Kernel Version: 4.4.52+
OS Image: Container-Optimized OS from Google
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://1.11.2
Kubelet Version: v1.6.7
Kube-Proxy Version: v1.6.7
PodCIDR: 10.24.0.0/24
ExternalID: 8419821342083849481
Non-terminated Pods: (7 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system fluentd-gcp-v2.0-t3rzf 100m (10%) 0 (0%) 200Mi (5%) 300Mi (8%)
kube-system heapster-v1.3.0-3440173064-d66jq 138m (13%) 138m (13%) 301456Ki (7%) 301456Ki (7%)
kube-system kube-dns-1829567597-n6kz6 260m (26%) 0 (0%) 110Mi (2%) 170Mi (4%)
kube-system kube-dns-autoscaler-2501648610-88ch6 20m (2%) 0 (0%) 10Mi (0%) 0 (0%)
kube-system kube-proxy-gke-wordpress-default-pool-91c14317-jdlj 100m (10%) 0 (0%) 0 (0%) 0 (0%)
kube-system kubernetes-dashboard-490794276-93cn2 100m (10%) 100m (10%) 50Mi (1%) 50Mi (1%)
kube-system l7-default-backend-3574702981-509zt 10m (1%) 10m (1%) 20Mi (0%) 20Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
728m (72%) 248m (24%) 700816Ki (18%) 854416Ki (22%)
Events: <none>
The config file (production.yaml):
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
spec:
replicas: 1
selector:
matchLabels:
app: wordpress
template:
metadata:
labels:
app: wordpress
spec:
terminationGracePeriodSeconds: 30
containers:
- image: wordpress:latest
name: wordpress
imagePullPolicy: "Always"
env:
- name: WORDPRESS_HOST
value: localhost
- name: WORDPRESS_DB_USERNAME
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: username
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
- image: nginx:latest
name: nginx
ports:
- containerPort: 80
name: nginx
- image: gcr.io/cloudsql-docker/gce-proxy:1.09
name: cloudsql-proxy
command: ["/cloud_sql_proxy", "--dir=/cloudsql",
"-instances=inspiring-tower-99712:europe-west1:wordpressdb=tcp:3306",
"-credential_file=/secrets/cloudsql/credentials.json"]
volumeMounts:
- name: cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
- name: ssl-certs
mountPath: /etc/ssl/certs
- name: cloudsql
mountPath: /cloudsql
volumes:
- name: wordpress-persistent-storage
gcePersistentDisk:
pdName: wordpress-disk
fsType: ext4
- name: cloudsql-instance-credentials
secret:
secretName: cloudsql-instance-credentials
- name: ssl-certs
hostPath:
path: /etc/ssl/certs
- name: cloudsql
emptyDir:
338
To find the cluster IP address of a Kubernetes pod, use the kubectl get pod command on your local machine, with the option -o wide . This option will list more information, including the node the pod resides on, and the pod's cluster IP. The IP column will contain the internal cluster IP address for each pod.
You can add the nodeSelector field to your Pod specification and specify the node labels you want the target node to have. Kubernetes only schedules the Pod onto nodes that have each of the labels you specify. See Assign Pods to Nodes for more information.
A Node can have multiple pods, and the Kubernetes control plane automatically handles scheduling the pods across the Nodes in the cluster.
There are additional containers running besides the ones specified in the config. These are provisioned by default with Kubernetes and run in the kube-system namespace which are not shown in the default namespace.
You can view all the pods by kubectl get pods --all-namespaces.
These additional containers are taking up 72% of the CPU quota of the single node...
Hence the 3 containers at 10% cpu quota would exceed 100% of the CPU quota (because 72% + (3 * 10) > 100%)...
As to why 72% of the containers is being allocated to the other containers - the question is asked here: Why does a single node cluster only have a small percentage of the cpu quota available?
Additional resources that may be usefull: How to reduce CPU limits of kubernetes system resources?
However, I was able to get the containers to run with sufficient CPU by adding additional nodes to the cluster. In addition, the high cpu instances seem to be more efficiently allocated on Google Cloud.
51
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
