rrd_reports.cgi
rrd_reports.cgi
Background
HTML and JSON reports are generated from various applications, including process_rrd_3 and FIDO. Historically, these reports were dumped into a directory and viewed by examining the contents of a directory, "ls" style.
In 2014, I began archiving these reports so you could go back in time to see reports generated on a given day. This was great, but left us with a mountain of data to pour through. In 2015, an sql backend and web front end were introduced to ease the burden of finding interesting things to look at.
HTML and JSON reports are generated from various applications, including process_rrd_3 and FIDO. Historically, these reports were dumped into a directory and viewed by examining the contents of a directory, "ls" style.
In 2014, I began archiving these reports so you could go back in time to see reports generated on a given day. This was great, but left us with a mountain of data to pour through. In 2015, an sql backend and web front end were introduced to ease the burden of finding interesting things to look at.
What/where
as59: https://stats.net.wisc.edu/cgi-bin/rrd_reports.cgi
as59: https://stats.net.wisc.edu/cgi-bin/rrd_reports.cgi
as3128: https://stats.uwsys.net/cgi-bin/rrd_reports.cgi
rrd_reports.pl is a front end to process_rrd_3 for generating reports. Daily, weekly and monthly reports are generated via net cron, but the script can also be ran by hand.
Both of the above read a config file in /usr/local/rrd_reports/etc/.
The SQL table is searchable via the following example URL, which is linked off the main stats page.
The rotation of files in /var/www/html/reports/archived is handled by the ns-rrd-reports project, specifically /etc/cron.d/ns-rrd-reports-purge-old-files.cron. Configuration for how many files to keep on rotation is kept in rrd_reports.pl.yaml sitelocal_config. If you update this file, you must update_config.pl and commit.
More info about rrd_reports.cgi
* 'items' is supposed to be a simple count of how many things are in the report.
* 'errors' is an indicator if the generating script had errors during generation, not to be confused with 'items'
* Type includes syslog severity link levels: [error, warning, info]. type=unknown is reserved for missing or non relevant types.
Table cell has a colored border legend:
rrd_reports_update.pl is daemonized and looks for incoming html reports. It's role is to archive incoming files appropriately and update the SQL backend. Incoming reports don't need to come from process_rrd_3 but need to have magic comment lines in order to take full advantage of the SQL/web frontend. Script can be ran non interactively via CLI to process files in batch or to scrub old/dead/broken files from disk.
Both of the above read a config file in /usr/local/rrd_reports/etc/.
The SQL table is searchable via the following example URL, which is linked off the main stats page.
The rotation of files in /var/www/html/reports/archived is handled by the ns-rrd-reports project, specifically /etc/cron.d/ns-rrd-reports-purge-old-files.cron. Configuration for how many files to keep on rotation is kept in rrd_reports.pl.yaml sitelocal_config. If you update this file, you must update_config.pl and commit.
Example:
[@stats1-cssc sitelocal_config]$ cat /usr/local/ns-rrd-reports/etc/sitelocal_config/rrd_reports.pl.yaml
---
paths:
incoming_dir: /var/local/ns-rrd-reports/incoming
rotate:
daily: '45'
monthly: '13'
weekly: '28'
More info about rrd_reports.cgi
* 'items' is supposed to be a simple count of how many things are in the report.
* 'errors' is an indicator if the generating script had errors during generation, not to be confused with 'items'
* Type includes syslog severity link levels: [error, warning, info]. type=unknown is reserved for missing or non relevant types.
Table cell has a colored border legend:
- green = no errors, no matches to report. Everything is great
- yellow = report has items to investigate
- red = the report generator itself had an error. Investigation recommended
- brown = linked html report is stale; either the test has been disabled or there is a problem with the report generation
- gray = the report didn't provide information about how many items were encountered. Recommended to update the report generator to include this information