FIDO: File Formats

FIDO: File Formats

Description of the Status Files written by FIDO tests

Want to make your own test? Here is the format of the status files.
All tests, including FIDO proper itself, will write out a status file that contains information about the test itself and objects related to the test.
<item data> 
   <failed-object [object name]> # appended with the test type to avoid collisions

	candidate
	       # CALCUATED BY FIDO: Only relevant for fido_node alarms.  List of contributing alarms
	       item_data s-hotspare-b360-17-hotspare-fido_node candidate [s-hotspare-b360-17-hotspare.net.wisc.edu-snmp_node]
        correlation 
	       # OPTIONAL
               item = [string]    # this object is related to an upstream object
               reason = [string]   # text description of why this correlation was made
        deleteme
		# OPTIONAL: [boolean]
                # delete this from the FIDO report file even if I didn't repoll a status update for everything this time around
        deleteme_instance
		# OPTIONAL: [interger]
                # delete this from the FIDO report file even if I didn't repoll a status update for everything this time around, but only this instance
	descr
		# CALCULATED BY FIDO: [string]: FIDO's calculated description of an item
        device
		# OPTIONAL string.  Used by the correlation engine

	event_id
		# CALCULATED BY FIDO: [number]: unique identifier for this event

	failures
		# CALCULATED BY FIDO: [integer]: number of times the update was determined to have failed

	file
		# CALCULATED BY FIDO: [string]: ingress filename for given alarm
help # OPTIONAL: will be displayed in the gui. If you don't supply, the test filename will be used key1 = help file name [string name] key2 = help file number [integer] holddown # CALCULATED BY FIDO: matches = [hash] # describes why the item in on holddown reason = [string] time = [integer] # time in minutes to hold down alarm based on alarm start time absolute_time = [string] # time in text to hold down alarm.
        impact
		# OPTIONAL: assumed 2 if not present
		value = integer
		reason = OPTIONAL [string]
        info
		# OPTIONAL [string]. gui mouseover or 'wideinfo' shows this information
	infohash
		# OPTIONAL:  successor of 'info' key

		examples: 
		infohash additional_info [122 accepted, 25 dropped, 1000 threshold]
		infohash icmp_source [fido_icmpv4_manual]
		infohash interface [port-channel109]
		infohash status [No response from '146.151.144.148'
		infohash timestamp [2020/04/01 09:20:00]
		infohash AS [65011]
		infohash Descr [Goodnight Hall/Communicative Disorders]
		infohash IP [192.168.1.1]
        interface
		# OPTIONAL [string] used by the correlation engine
        ip
		# OPTIONAL [ip address] used by the correlation engine and also used for DNS lookups

	ipam
		# OPTIONAL: CALCULATED BY FIDO: 
		ipam circuit_id [value undefined]
		ipam connector [value undefined]
		ipam subnet [146.151.144.148/32]
        iso
		# [integer] used for correlation.  The lower the value, the more important to the correlation engine.
		example, ip=3, service=4, so service is correlated to IP, not vice versa.
        keepme
		# OPTIONAL: [boolean]
                # keep this in the FIDO report file even if it isn't in the current status file
                # this is to support objects that aren't polled every cycle

	logged
		# CALCULATED BY FIDO: [boolean]: whether the item has been logged
 
        needed_failures
		# OPTIONAL [integer]: defaults to one.  number of consecutive poll failures before the object shows in the gui

        rtt
		# OPTIONAL [number] supplied by some tests to describe how long it took to verify this particular alarm. 

	snmp_instance
		# OPTIONAL [integer]: for debugging

	source
		# CALCULATED BY FIDO.PL: federation information 
		source fido-animal.net.wisc.edu [1]
		source fido-cssc.net.wisc.edu [1]
start # CALCULATED BY FIDO.PL: time in ticks alarm was first seen start_text # CALCULATED BY FIDO.PL: time in string alarm was first seen
        state
		# REQUIRED: [string]: state of the tested object
	status
		# CALCULATED BY FIDO.PL: whether or not the item is stale, ok, etc

        suppressed
		# OPTIONAL: [integer]: when suppression is being used, how many items are not shown because of this parent item.  Supplied by fido_snmp.pl

        test
		# REQUIRED: [string], the type of failure, displayed in the 'type' column, often used for correlation

        time
		# REQUIRED: [interger]: time in ticks that the object failed during the current test period
       time_of_day
		# CALCULATED BY FIDO: OPTIONAL
               matches = [hash]
                   # describes why the item in on holddown
               actual_match = [string]
               reason = [string] 
               time = [string]
                   # time in text to hold down alarm.
        ts_device
		# OPTIONAL string.  Used by the correlation engine
        ts_urls
		# OPTIONAL: key/value of url_key and destination
		ts_urls gnmis [http://stats.net.wisc.edu/cgi-bin/gnmis.fcgi?device=^s-hotspare-b360-17-hotspare.net.wisc.edu]
	updated
		# CALCULATED BY FIDO: [boolean]: 
		This is used to determine if an object should be removed from the report file.  
		The item is removed if the item itself isn't updated but the file it came from has been. 
urls # CALCULATED BY FIDO: key/value of url_key and destination urls __alarm_info [https://fido-cssc.net.wisc.edu/cgi-bin/fido_alarm_info.cgi?item=2607%3Af388%3A2%3A6002%3A%3A3-icmpv6&amp;test=icmpv6] urls rrd [<a href=http://stats.net.wisc.edu/cgi-bin/genstatspage.fcgi?rrdfile=/data/mrtg/icmpv6/ip=2607:f388:2:6002::3_family=ipv6_icmpv6.rrd> cgi: rrd <IMG SRC="http://stats.net.wisc.edu/cgi-bin/gengraph.fcgi?rrdfile=/data/mrtg/icmpv6/ip=2607:f388:2:6002::3_family=ipv6_icmpv6.rrd"></a>]
	via_connected
		# CALCULATED BY FIDO: correlation
		item_data 2607:f388:e:101::10-icmpv6 via_connected subnet [2607:f388:e:101::1/64]

< /item data > 

grep "added: item_data" output.txt | egrep -v " (correlation|device|help|impact|info|infohash|interface|ip|iso|needed_failures|state|suppressed|test|time|url|start|start_text|status|source|event_id|failures|file|logged|urls|updated|descr|ipam|ts_device|ts_urls|candidate|via_connected|snmp_instance|deleteme|deleteme_instance) " 


< status >
        alldown
		# OPTIONAL: [integer]: false if everything isn't down, otherwise it will be the number of down items

	contributors
		# OPTIONAL: unique to updateType [scraper], lists health of downstream contributors
		contributors ban-arp state [OK]
		contributors fp-arp state [OK]
		contributors fp-systemEnvironmentals state [OK]

	data_files
		# OPTIONAL, but really good idea: source data to show in fido_info.cgi
		data_files gMegaFlow_fido_icmp_ips.bin []
		data_files gMegaFlow_fido_icmp_ips_verbose.bin []

        items
		# REQUIRED: [string] : total monitored items, not a count of what's down
		dhcp_leases items [820 rrd files [731 file(s) ignored]]

        last_cycle_time
		# REQUIRED: [integer] (in seconds)

	last_poll
		# OPTIONAL: unique to updateType [scraper], informational/debug
        last_update
		# OPTIONAL: [integer] (in seconds), last time a data source was updated
polled # OPTIONAL [array of polled items]. A list of 'test' types that had been updated during this most recent cycle. For tests that may not be testing every test type all of the time fido.status: added: snmprrd polled ADVAopticalIfDiagInputPower [array_string] fido.status: added: snmprrd polled ADVAopticalIfDiagOutputPower [array_string] fido.status: added: snmprrd polled AlphadataNumberValue [array_string] ... report # REQUIRED: [integer]: always increases by one, make sure we don't miss a report
        report_time
		# REQUIRED: [integer]: time this report was written
        restart
		# OPTIONAL: [string]: used in conjunction with 'user' for fido_watchdog
        stale
		# REQUIRED: [integer]: seconds after last update before this data is stale 
start # OPTIONAL: [integer]: ticks when this daemon started status # CALCULATED BY FIDO: [string]: OK, stale, etc threads # OPTIONAL: [integer]: how many thread the test is configured to use updateType # OPTIONAL: [string]: scraper_alarm has a unique update methodology. This is a trigger to fido.pl to behave differently url # OPTIONAL: [string]: points to file that contains a list of items, display in the FIDO status page uc_esx url [/uc_esx.txt] user # OPTIONAL: [string]: to be used by fido_watchdog < /status > status_output.txt | egrep -v " (items|last_cycle_time|polled|report|stale|threads|url|status|report_time|last_update|data_files|user|start|restart|contributors|updateType|last_poll) "

Description of fido.report.dat

The main report file that is read by the CGI
   'comment_correlation' =>
        'comment' => # a comment
             'best' => # the best item for this comment, as determined by FIDO.
                       # best is defined by being the item with the most things
                       # correlated to it.  Other tiebreakers are also used.
             'items' =>  # a list of items with the above comment
                 'item with the comment mentioned above'

	comment_correlation 'cr 90035' best [s-bocklabs-254-1-access.local.net.wisc.edu ifOperStatus Gi1/0/1-ifOperStatus]
	comment_correlation 'cr 90035' items 's-bocklabs-254-1-access.local.net.wisc.edu ifOperStatus Gi1/0/1-ifOperStatus' [1]
	comment_correlation 'cr 90035' items 10.151.40.134-icmpv4 [1]   

   'comments' => # where the comments are kept
                 # they are kept separate from the items themselves as we keep comment information for items that get deleted.
	# comment for this item
	comments '$itemName' comment [Problem 15824]

	# old comment data gets turned into a hash array and appended when the comment changes
	comments '$itemName' comment_history [{"discovery":"pending","first_seen":1578300837,"occurrences":1,"related_event_id":"20200106.3575"},{"discovery":"1575917448","first_seen":1578300837,"occurrences":1,"related_event_id":"20200106.3575"},{"comment":"Problem 15824","date":1578317204.17587,"discovery":"1575917448","first_seen":1578300837,"occurrences":1,"related_event_id":"20200106.3575","user":"aamohamed3"}] [array_jsonified]
	# [integer, ticks]; When this comment was last updated
	comments '$itemName' date [1578317204.17587]
# when the item was first monitored by FIDO comments '$itemName' discovery [1578410731]
	# [integer, ticks]; time the alarm was added
	comments '$itemName' first_seen [1578300837]
#[integer, ticks]; time the alarm was removed comments '$itemName' last_seen [1584513560]
	#[integer, ticks]; length in seconds that this alarm has been historically alarming.  Only present for items that have been removed at least once
	comments '$itemName' length [6200691]
	#[integer]; number of times the alarm has been seen
	comments '$itemName' occurrences [115]
        # can be used to link to an event in the sql log DB [if applicable]
	comments '$itemName' recurring_event_id [20200303.50091]
# can be used to link to an event in the sql log DB comments '$itemName' related_event_id [20200106.3575] # who commented the item comments '$itemName' user [aamohamed3]
   'correlated'
	# summary of known network topology correlations as determined by fido.pl
	examples:
	correlated 'r-wa222-12-2-radial.net.wisc.edu ifOperStatus Vl2671-ifOperStatus' 146.151.178.213-icmpv4 [1]
	correlated 's-chmbrln-1327-1-radial.local.net.wisc.edu ifOperStatus Po7-ifOperStatus' 's-chmbrln-1327-1-radial.local.net.wisc.edu ifOperStatus Te1/0/7-ifOperStatus' [1]
	correlated 10.151.129.46-icmpv4 2607:f388:f:1c::2-icmpv6 [1]
	correlated s-hotspare-b360-17-hotspare-fido_node s-hotspare-b360-17-hotspare.net.wisc.edu-snmp_node [1]
	correlated s-hotspare-b360-17-hotspare.net.wisc.edu-snmp_node 146.151.144.148-icmpv4 [1]
'event_id' [string] #next event will be assigned this ID event_id [20200401.31941]
   'fido_start_time'
	[integer, ticks]
	fido_start_time [1585746923]
   'group_correlated'
	# hash structure defining correlation groupings based on alarm attributes
	group_correlated 2607:f388:2:6002::3-icmpv6 group [group='ipam descr=DoIT Platform Non Restricted Data']
	group_correlated 2607:f388:2:6002::3-icmpv6 members 2607:f388:2:6002::2-icmpv6 [1]
   'last_cycle_time' => [string]
       	# how long it took to create the last fido report
	last_cycle_time [<1]
'report' => [integer] # fido report number, increases by one every time a new report is written. report [220]
   'report_time'
	[integer, ticks]
	When the report was written
	report_time [1585751633]
}




Keywords:FIDO: File Formats   Doc ID:9171
Owner:Michael H.Group:Network Services
Created:2009-02-17 19:00 CDTUpdated:2020-04-02 11:20 CDT
Sites:Network Services, Systems & Network Control Center, University of Wisconsin System Network
Feedback:  1   0