Reports

Reports

Background

HTML and JSON reports are generated from various applications, including process_rrd_3 and FIDO.  Historically, these reports were dumped into a directory and viewed by examining the contents of a directory, "ls" style.

In 2014, I began archiving these reports so you could go back in time to see reports generated on a given day.  This was great, but left us with a mountain of data to pour through.  In 2015, an sql backend and web front end were introduced to ease the burden of finding interesting things to look at.


What/where

rrd_reports.pl is a front end to process_rrd_3 for generating reports.  Daily, weekly and monthly reports are generated via net cron, but the script can also be ran by hand.

rrd_reports_update.pl is daemonized and looks for incoming html reports.  It's role is to archive incoming files appropriately and update the SQL backend.  Incoming reports don't need to come from process_rrd_3 but need to have magic comment lines in order to take full advantage of the SQL/web frontend.  Script can be ran non interactively via CLI to process files in batch or to scrub old/dead/broken files from disk.  

Both of the above read a config file in /usr/local/rrd_reports/etc/. 

The SQL table is searchable via the following example URL, which is linked off the main stats page.

https://stats.uwsys.net/cgi-bin/rrd_reports.cgi
https://stats.net.wisc.edu/cgi-bin/rrd_reports.cgi


More info about rrd_reports.cgi

Magic variables to include in the source file are as follows [as of 2015/04/27]

     if (m/^#matches=(\d+)$/) {
        $attributes{'items'} = $1;
     } elsif (m/^#errors_encountered=(\d+)$/) {
        $attributes{'errors'} = $1;
     } elsif (m/^#Files Examined=(\d+)$/) {
        $attributes{'considered_items'} = $1;
     } elsif (m/^#Start='(.+) -/) {
        $attributes{'start'} = $1;
     } elsif (m/^#End='(.+) - /) {
        $attributes{'end'} = $1;
     } elsif (m/^#Description=(.+)/) {
        $attributes{'description'} = $1;
     } elsif (m/^#report_type=(.+)/) {
        $attributes{'type'} = $1;
     } elsif (m/^#processing_took=(\d+) /) {
        $attributes{'runtime'} = $1;
     }

* 'items' is supposed to be a simple count of how many things are in the report.
* 'errors' is an indicator if the generating script had errors during generation, not to be confused with 'items'
* Type includes syslog severity link levels: [error, warning, info].  type=unknown is reserved for missing or non relevant types.

Table cell has a colored border legend:
  • green = no errors, no matches to report.  Everything is great
  • yellow = report has items to investigate
  • red = the report generator itself had an error.  Investigation recommended
  • brown = linked html report is stale; either the test has been disabled or there is a problem with the report generation
  • gray = the report didn't provide information about how many items were encountered.  Recommended to update the report generator to include this information




Keywords:Reports rrd_reports.cgi rrd reports   Doc ID:50855
Owner:Michael H.Group:University of Wisconsin System Network
Created:2015-04-27 07:26 CSTUpdated:2017-09-13 07:38 CST
Sites:Network Services, Systems & Network Control Center, University of Wisconsin System Network
Feedback:  0   0