Dtrace script that returns %CPU, %Memory and network in/out aggregated by zone

Question

I need a DTrace script that returns CPU usage percentage, memory usage percentage and network input and output bytes used, aggregating the data by zone. Something like this:

ZONE  %CPU %MEM NET_INPUT NET_OUTPUT
zone1 25%  12%   86012     1294
zone2 48%  65%   86012     7354
zone3  8%  14%  268153    68746

By now, I am using:

prstat -Z -c -n 1,99999 5 1

That return something similar (without network traffic):

Please wait...
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
 32051 root     1090M 1079M sleep    1    0  12:57:25 0,4% qemu-system-x86/7
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE                        
  3228        2 1090M 1079M   2,2%  12:57:25 0,4% 6ce064b6-fec9-4daa-ba2b-0082f73fca73
     0      105 2075M  633M   1,1% 202:56:51 0,4% global                      
  3031        2 2105M 2094M   4,3%  17:04:52 0,3% e32fb987-35f7-4860-a04a-ca26c327d4ba
(...)
  3411       18   81M   54M   0,1%   0:00:04 0,0% 52120eb6-2e20-4a64-8f7a-235a44d9f100
Total: 894 processes, 5460 lwps, load averages: 0,53, 0,53, 0,54

I need to get the same, but done with DTrace, in order to combine the data with network traffic, and more probes in the future.

Answer 1

The reason such a script doesn't already exist is that CPU / memory utilization is pretty difficult to get using DTrace. DTrace is best for sampling data on a certain event. To get CPU utilization using DTrace, you would need to track every time a CPU became free and every time it became busy and then do some addition. Note that this is different from the provider that DTrace gives for tracking scheduler operations since those are on a per-thread basis rather than a per-CPU basis. Memory is even more annoying, since you would be tracking every memory allocation and deallocation.

To get these data sources, you might be better served by pulling the data from kstat (it sounds like you're using Solaris, which AFAIK is the only platform that has kstat). The information you're looking for can be found using it like so:

$ sudo kstat unix:0:system_misc:ncpus      # this is the number of CPUs you have
module: unix                            instance: 0     
name:   system_misc                     class:    misc
        ncpus                           2

$ sudo kstat cpu::sys:cpu_ticks*           # ticks of each type for each core since boot
module: cpu                             instance: 0     
name:   sys                             class:    misc
        cpu_ticks_idle                  9375292
        cpu_ticks_kernel                82658
        cpu_ticks_user                  23684
        cpu_ticks_wait                  0

module: cpu                             instance: 1     
name:   sys                             class:    misc
        cpu_ticks_idle                  9410367
        cpu_ticks_kernel                49141
        cpu_ticks_user                  21956
        cpu_ticks_wait                  0

$ sudo kstat unix:0:system_pages:physmem   # pages of physical memory (multiply by page size for number of bytes)
module: unix                            instance: 0     
name:   system_pages                    class:    pages
        physmem                         1045390

$ sudo kstat unix:0:system_pages:freemem   # pages of free memory (multiply by page size for number of bytes)
module: unix                            instance: 0     
name:   system_pages                    class:    pages
        freemem                         880842

Note that you need to subtract each new reading from the last reading for the CPU ticks counters - otherwise, you'll be tracking the total number of ticks since system boot. When the counters overflow, they are set to 0 and then the new value is added to them (it's not just a blind addition overflow).

You can also use kstat to monitor the number of bytes read/written over network links using the link:0::. I'm not sure if this information can be found per-zone, but there's probably a kstat that tracks that, too.

I guess if you were determined to use DTrace to monitor the values, you could force it to do so by tracking the moments when these counters are being modified in the kernel and record the modifications. However, I don't really see the point of doing that as it's easier to use other methods as you've already discovered. Why not make a script that starts up both data sources and combine the results into something nicer to look at?

Answer 2

Take a look at Brendan Gregg's DTraceToolkit. It has a huge collection of scripts that uses various probes to accomplish a wide set of tasks. That might point you in the right direction.