rrd_reports.cgi

rrd_reports.cgi

Background

HTML and JSON reports are generated from various applications, including process_rrd_3 and FIDO.  Historically, these reports were dumped into a directory and viewed by examining the contents of a directory, "ls" style.

In 2014, I began archiving these reports so you could go back in time to see reports generated on a given day.  This was great, but left us with a mountain of data to pour through.  In 2015, an sql backend and web front end were introduced to ease the burden of finding interesting things to look at.



What/where

as59: https://stats.net.wisc.edu/cgi-bin/rrd_reports.cgi
as3128: https://stats.uwsys.net/cgi-bin/rrd_reports.cgi

rrd_reports.pl is a front end to process_rrd_3 for generating reports.  Daily, weekly and monthly reports are generated via net cron, but the script can also be ran by hand.

rrd_reports_update.pl is daemonized and looks for incoming html reports.  It's role is to archive incoming files appropriately and update the SQL backend.  Incoming reports don't need to come from process_rrd_3 but need to have magic comment lines in order to take full advantage of the SQL/web frontend.  Script can be ran non interactively via CLI to process files in batch or to scrub old/dead/broken files from disk.  

Both of the above read a config file in /usr/local/rrd_reports/etc/. 

The SQL table is searchable via the following example URL, which is linked off the main stats page.

The rotation of files in /var/www/html/reports/archived is handled by the ns-rrd-reports project, specifically /etc/cron.d/ns-rrd-reports-purge-old-files.cron.  Configuration for how many files to keep on rotation is kept in rrd_reports.pl.yaml sitelocal_config. If you update this file, you must update_config.pl and commit.

Example:

[@stats1-cssc sitelocal_config]$ cat /usr/local/ns-rrd-reports/etc/sitelocal_config/rrd_reports.pl.yaml 
---
paths:
  incoming_dir: /var/local/ns-rrd-reports/incoming
rotate:
  daily: '45'
  monthly: '13'
  weekly: '28'




More info about rrd_reports.cgi

* 'items' is supposed to be a simple count of how many things are in the report.
* 'errors' is an indicator if the generating script had errors during generation, not to be confused with 'items'
* Type includes syslog severity link levels: [error, warning, info].  type=unknown is reserved for missing or non relevant types.

Table cell has a colored border legend:
  • green = no errors, no matches to report.  Everything is great
  • yellow = report has items to investigate
  • red = the report generator itself had an error.  Investigation recommended
  • brown = linked html report is stale; either the test has been disabled or there is a problem with the report generation
  • gray = the report didn't provide information about how many items were encountered.  Recommended to update the report generator to include this information






Keywords:Reports rrd_reports.cgi rrd reports   Doc ID:50855
Owner:Michael H.Group:University of Wisconsin System Network
Created:2015-04-27 07:26 CSTUpdated:2022-05-23 14:56 CST
Sites:Network Services, Systems & Network Control Center, University of Wisconsin System Network
Feedback:  0   0