GCE - No stackdriver memory metrics for nodes

Question

I have set up my Kubernetes 1.3.4 cluster on GCE with

export KUBE_ENABLE_CLUSTER_MONITORING=google

This works quite nicely, I get application logs (for some reason in the Container Engine section, but well) and also pod and node metrics.

The only thing that is missing are the node memory metrics, only CPU is shown (see screenshot)

No memory metrics

In the heapster logs I see tons of lines like this

{
 metadata: {
  severity: "ERROR"    
  projectId: "<project-id>"    
  serviceName: "container.googleapis.com"    
  zone: "europe-west1-d"    
  labels: {
   container.googleapis.com/cluster_name: "production"     
   compute.googleapis.com/resource_type: "instance"     
   compute.googleapis.com/resource_name: "fluentd-cloud-logging-production-minion-group-p0w8"     
   container.googleapis.com/instance_id: "6772154497331326454"     
   container.googleapis.com/pod_name: "heapster-v1.1.0-2102007506-23b3e"     
   compute.googleapis.com/resource_id: "6772154497331326454"     
   container.googleapis.com/stream: "stderr"     
   container.googleapis.com/namespace_name: "kube-system"     
   container.googleapis.com/container_name: "heapster"     
  }
  timestamp: "2016-09-13T14:40:08.000Z"    
  projectNumber: "930564692351"    
 }
 textPayload: "E0913 14:40:08.665035       1 gcm.go:179] Error while sending request to GCM googleapi: Error 400: Timeseries 76, point: start is not older than end, for a cumulative metric, invalidParameter
"   
 insertId: "pt5bo7g132r266"   
 log: "heapster"   
}

Not sure if this is related.

Any ideas?

Answer 1

If you are running your cluster using GCE instead of GKE You should install the stackdriver agent and verify the credentials that agent is using to communicate with stackdriver link

If you are using linux you can install the agent by executing:

curl -sSO https://dl.google.com/cloudagents/install-monitoring-agent.sh
sudo bash install-monitoring-agent.sh

and you can check your credentials running the following command:

sudo cat $GOOGLE_APPLICATION_CREDENTIALS
sudo cat /etc/google/auth/application_default_credentials.json