Incorrect reporting of container memory usage by cadvisor

Question

cAdvisor reports 14GB for the memory used by Prometheus where as top reports 6.xGB.

Can someone explain why this discrepancy?

The documentatation of container_memory_usage_bytes says

Current memory usage in bytes, including all memory regardless of when it was accessed

but it's not clear what this refers to - I assume it's virtual memory size?

As reported by cAdvisor:

core@ip-172-20-100-148 ~ $ curl -q localhost:4194/metrics | grep container_memory_usage_bytes | grep prometheus
container_memory_usage_bytes{container_name="prometheus",id="/docker/d37e7503309e632265cb834095efa949da4dc3c72122cb290e626f1121d0ed6b",image="quay.io/prometheus/prometheus:v1.7.1",name="k8s_prometheus.71f5f8c6_prometheus-zwvhx_xx-system_9753de21-aaac-11e7-97e5-026a05f72f78_4682ea0b",namespace="xx-system",pod_name="prometheus-zwvhx"} 1.4178545664e+10

As reported by top on the host machine:

Tasks: 272 total,   1 running, 271 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.2 us,  2.7 sy,  0.0 ni, 75.9 id,  0.5 wa,  0.6 hi,  0.1 si,  0.0 st
KiB Mem:  65964640 total, 64892592 used,  1072048 free,  5222520 buffers
KiB Swap:        0 total,        0 used,        0 free.  4737420 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                   
118446 root      20   0 6783108 6.154g  19864 S 316.7  9.8   8241:05 prometheus 

Answer 1

Per the discussion here, cAdvisor reports a total memory usage in bytes.

In your case, cAdvisor was reporting 1.4178545664e+10 Byte = 14.178545664 Gigabyte. Note it was e+10 so it is approximately 14G, not 1.4G.