custom Ganglia cluster graphs - how to show average metric value, rather than total metric value

Question

How do you show mean values in custom graphs, rather than total values?

Previously I've had success generating cluster summary graphs by creating custom .php files, as described here:

http://sourceforge.net/apps/trac/ganglia/wiki/Custom_graphs

However, up to this point, I've wanted to show total actions per second for a given metric. But now, I have some timing data where I want to show the mean (average) metric value for all cluster nodes. How is this done? With my current implementation, the generated graphs show the total time value for all nodes, which is not helpful.

Here is the .php:

<?php

/* Pass in by reference! */
function graph_jmx_times_report ( &$rrdtool_graph ) { 

    global $context,
           $hostname,
           $graph_var,
           $range,
           $rrd_dir,
           $size,
           $strip_domainname;

    if ($strip_domainname) {
       $hostname = strip_domainname($hostname);
    }   

    $jmx = $graph_var;
    $title = $jmx.' Processing Time';
    if ($context != 'host') {
       $rrdtool_graph['title'] = $title;
    } else {
       $rrdtool_graph['title'] = "$hostname $title last $range";
    }   
    $rrdtool_graph['lower-limit']    = '0';
    $rrdtool_graph['vertical-label'] = 'milliseconds';
        $rrdtool_graph['extras']         = '--rigid --base 1024';
    $rrdtool_graph['height'] += ($size == 'medium') ? 89 : 0;

        $series = "DEF:'tot_time'='${rrd_dir}/jmx_tomcat_proc_time_ms.rrd':'sum':AVERAGE"
                ."DEF:'fc_time'='${rrd_dir}/jmx_tomcat_freqcap_lookup_time_75.rrd':'sum':AVERAGE "
                ."DEF:'ro_time'='${rrd_dir}/jmx_tomcat_readonly_lookup_time_75.rrd':'sum':AVERAGE "
                ."DEF:'rt_time'='${rrd_dir}/jmx_tomcat_realtime_lookup_time_75.rrd':'sum':AVERAGE "
                ."AREA:'tot_time'#CFF1FC:'' "
                ."LINE2:'fc_time'#F19A2A:'Freq Cap 75' "
                ."LINE2:'ro_time'#66CC33:'Read-only 75' "
                ."LINE2:'rt_time'#CC99CC:'Realtime 75' "
                ."LINE2:'tot_time'#20ABD9:'Processing Time' "
        ;

    $rrdtool_graph['series'] = $series;

    return $rrdtool_graph;

}

?>

Answer 1

The summary RRDs Ganglia uses have a num data source in addition to the sum data source. The num value indicates the number of hosts the sum was calculated from and is available only in summary RRDs.

You can use the RRDtool CDEF instruction to calculate the average over all hosts by dividing sum by num.

Here's an example:

if ($context != 'host') { // cluster or grid summary graph context
    $series = 
     "'DEF:fc_time_sum=${rrd_dir}/jmx_tomcat_freqcap_lookup_time_75.rrd:sum:AVERAGE' " 
    ."'DEF:fc_time_num=${rrd_dir}/jmx_tomcat_freqcap_lookup_time_75.rrd:num:AVERAGE' " 
    ."'CDEF:avg_fc_time=fc_time_sum,fc_time_num,/' " 
    ."'LINE2:avg_fc_time#0000FF:Average Freq Cap 75' ";
} else { // single host graph context
    //here you can't use the num data source
}

This averaging is also shown in the sample_report.php file included with ganglia-web.