Topics Map > Campus Network
AANTS - Everything You Ever Wanted To Know About the NetWatch Tool
Background Information
The AANTS NetWatch tool displays MAC address and IP information for device ports on the XXI network. The tool tries to show the IP address that was most recently seen associated with a given MAC address. The NetWatch database tables record the MAC addresses seen on a given port, and the IP-to-MAC relationship observed (at a different point in time) as learned from the router ARP tables.NetWatch's notion of the current MAC address in use on a given port, is updated in two ways:
1) By periodic polling
2) Whenever the MAC address is learnt, the switch is configured to send it to our service (as an SNMP trap).
So, when you plug a machine onto a port, and that machine then generates an ethernet frame, AANTS (and EdgeConf forms which display it) should show that new value immediately.
However, if a machine is continually producing traffic, then the switch will not relearn the address (since it already knows that that MAC is on the given port) so, AANTS will only periodically re-discover that address.
The physaddrwatch tool is run four times an hour by cron:
0,15,30,45 * * * * /usr/bin/perl -e '$_ = (getpwnam($ENV{"LOGNAME"}))[8] ||
die "getpwnam: $!"; m:^(.*/(\w+))$: || die "match failed"; exec {$1} "-" . $2, "-c",
"cd /var/local/watch/physaddrwatch && make -s ngrun"; die "exec: $!"'
However, this has been slowing as more switches have been added to the new campus network.
If IP or MAC addresses show in the NetWatch report, then we reliably know they have been on the network at least at the dates shown when you click "with dates", but it does not necessarily mean that they haven't been on the network if they weren't learned.
NetWatch only knows the IP to MAC relationships that it learns from the DoIT-managed routers that it monitors. Users will not be able to use NetWatch to find information on IP addresses that are not routed on the XXI campus network routers. This would be the case, for instance, if you are a "delegated" customer.
If this is the case, you can find the MAC address on the gateway router then use NetWatch to search by MAC address to find the switchport(s) on which that MAC has been seen, if it has been seen on DoIT-managed switches.
NetWatch Q&A
Q: How do I do a query on a specific MAC address?
Q: I've entered a MAC address in the NetWatch page, but the only result I get is a blank page. Why?
Q: I've entered a MAC address in the NetWatch page, but the results seem out of date. Why?
Q: Why are multiple rows returned for a single device port?
Q: I've tried querying NetWatch for IP addresses I know are connected, but no results are returned. What gives?
Q: When I input an IP I see several MAC addresses, but when I enter one of those MAC addresses I only get one IP. What gives?
Q: Is there a way to force NetWatch to update its view of the world?
Q: I think some information in the NetWatch database is outdated. How can I confirm what's really going on?
Q: Is there a way I may find which enabled ports have been "inactive for N days" on a switch?
Q: Is it possible to query all managed switch ports that have multiple mac addresses on them?
Q: Why is my MAC address displayed with '127.0.0.1' as the IP address?
Q: Why do I get a 'Target Unknown' error when I click on 'Bits', 'Pkts' or 'Errs' for a particular port?
Q: What's with the weird addresses in the NetWatch database?
Q: Does the NetWatch database have information from csscplat devices?
Q: What about wireless data?
Q: How do I do a query on a specific MAC address?
Q: I've entered a MAC address in the NetWatch page, but the only result I get is a blank page. Why?
Q: I've entered a MAC address in the NetWatch page, but the results seem out of date. Why?
A: There are many reasons this might be the case:
If no results are returned it means that the MAC was not found using the given selection criteria. Try the query again, but this time click "with unknown IP addresses". This causes NetWatch to show where the MAC address was seen, even if it doesn't know what IP address it was using. If that doesn't yield anything, try a wider time range, like 9999 days rather than past 5 days (which is the default).
Another reason that MAC addresses aren't learned is that the switch port is incorrectly configured. The following lines are needed in the config to generate MAC address traps (which our system subsequently learns):
snmp trap mac-notification change added snmp trap mac-notification change removed
Additionally, the following lines should be configured on the device itself:
mac address-table notification change interval 300 mac address-table notification change history-size 500 mac address-table notification change
XXI engineers can look for mis-configured ports this by using CodePusher to run "show mac-address-table dynamic" on all devices matching "^[rs]-", then examining the logs afterward to find any non-uplink port that has learned that MAC. If your department runs its own router for IP hosts in that VLAN, you should be able to take that MAC address, and log in on your router for that LAN, and look in its ARP cache to see what IP address it says it's using.
MAC or IP information may not be learned by the NetWatch tool if there is a user firewall device between our gear and yours. Some firewall devices essentially rewrite the MAC address to its own. In these cases, the DoIT router is not really the "first-hop" gateway router for a user's LAN, and therefore can't see which of their LAN stations' MACs are actually using a given IP address. In these cases there is no way for NetWatch to show the machine's IP on a given switchport because the only device that "knows" the MAC-to-IP relationship is the customer-owned firewall. Other firewalls (such as the netscreens with which I'm familiar) can operate in a "bridge" mode in which they enforce policies without rewriting the MAC addresses. Perhaps another mode for your firewall also works in this way, but as currently configured we can just show MACs for your LAN. In NetWatch then, you should always choose the checkbox "with unknown IP addresses".
There is a problem with the physaddrwatch script (which discovers the MAC to IP relationships) that if any of the routers it tries is not reachable, the script terminates and therefore doesn't learn the MAC to IP relationship for any routers that occur alphabetically after the one that failed (such as "r-pete..."). If the device in question is added to the NETWORK_DEVICES in the firewall configuration things should start working.
Q: Why are multiple rows returned for a single device port?
A: The key to understanding the NetWatch report when it returns multiple rows is to pay attention to the dates in the "MAC seen" and "IP seen" columns. The most recent date is likely the most interesting, but other dates can give you historical information about where the MAC and IP addresses have been seen on the network.
Q: When I input an IP I see several MAC addresses, but when I enter one of those MAC addresses I only get one IP. What gives?
A: We don't show older IP address results for MAC addresses because they could be wrong... we don't know that that was the IP address that the given MAC used on that port at the time that that MAC was observed on that port. The two pieces of information are gathered independently. We show the most recently known MAC to IP translation for the MAC specified (unless an IP was specified) in the query. It's shown with a seperate date "IP Seen" to indicate that this is not the date the MAC was seen on that port, just for informational purposes. For static MAC to IP relationships this is more likely to be useful, less so for dynamic addresses.
When you search by MAC (only), one should read those results as "Here are the known port locations for MAC 00235ac7f9ae" and separately "That MAC has been most currently seen using IP 128.104.192.5."
Our database schema doesn't store IP addresses on Ports. Instead it stores only what the switches know (which MAC is on which port) and what the routers know (which IP is associated with which MAC). Since these are observed independently, we don't have any confidence in doing some temporal join across those two pieces of info. So, we fall back to just showing the most current. As it is now, in the typical query, showing recent data, we can be fairly sure that the two most recent observations can considered to coincide, that is why NetWatch "wants" to show the most recent observations of the MAC and IP (at the top), but one still must think about the "Seen" dates to determine whether it is likely that the machine on that port is using the IP shown.
Q: I've tried querying NetWatch for IP addresses I know are connected, but no results are returned. What gives?
A: Try choosing your devices, but don't enter any other selection criteria (don't enter the IP range). That will show you an "inventory" of what NetWatch thinks is connected on each port of those switches. As stated above, if you have a firewall device which is blocking MAC information, or if your gear is not routed on XXI network gear, there may be limitations on our ability to learn your MAC and/or IP information.
Q: Is there a way to force NetWatch to update its view of the world?
A: Not via the web interface. This would require polling the switch via SNMP, which we already do continually, in round-robin fashion, across all switches on campus. Technically you could force an update "in the field" by unplugging and replugging the machine's ethernet connection. The switch should then relearn the MAC, and send the notification to our server which will put it in the AANTS NetWatch database.
Q: I think some information in the NetWatch database is outdated. How can I confirm what's really going on?
A: If you suspect information in the NetWatch database does not reflect the current state of the world, you can query the switch directly yourself using the AANTS "Looking Glass" tool. Choose the Device then the Query named "show mac-address-table" or "show mac-address-table interface" (and specify the interface) and it will query the switch on demand.
Q: Is there a way I may find which enabled ports have been "inactive for N days" on a switch?
A: Yes, you can get an "inventory" of active ports for your switch using NetWatch, showing the last date when a MAC address was learnt. To get this report, just choose the switch name, and Submit Query.
You can tell it to only show you ports that have had activity at some time, but not in the past `n' days.
To do this, on the NetWatch form:
1. Choose the switch
2. Select "Report entries not seen within the past [ ] days" and fill in the number of days, e.g. 30
3. Leave other options alone.
4. Click Submit Query query.
When you are trying to determine the last time that port was considered active, be sure to pay attention to the "MAC not seen here since" date rather than the "IP seen" date. (The "IP seen" date is mostly irrelevant for your purpose since it is the time that IP was in use by that MAC as seen by the router... so it may have been on a different switchport then.)
Q: Is it possible to query all managed switch ports that have multiple mac addresses on them?
A: Not exactly. You can, however, run the existing NetWatch report (for one switch at a time) and look down the list of sorted port names, to multiple occurances of a port name. Those are the ports that had multiple MACs within the time range you supply in the "Report entries seen within the past [5] days" field.
Q: Why is my MAC address displayed with '127.0.0.1' as the IP address?
A: Showing the IP as 127.0.0.1 is not incorrect. That is the localhost IP address that exists inside every IP host. NetWatch displays it that way to get the SQL query to work when there is no discovered routable IP address.
Q: Why do I get a 'Target Unknown' error when I click on 'Bits', 'Pkts' or 'Errs' for a particular port?
A: It is possible that the port has not been in use long enough for the measurement system to start collecting data for that port. Usually our measurement system will start measuring the port the first morning after a MAC address is learned on the given port.
Q: What's with the weird addresses in the NetWatch database?
| 7719.99 | | 998.225 |They're not standard IP addresses which is what the rest of the table seems to be.
A: These are left-over AppleTalk addresses. The SNMP table on the router has various layer-3 addresses associated with a give MAC, depending on what protocols the router is configured to route. Mostly, today, we're just concerned with IP addresses, of course. (I don't think AppleTalk nor IPX routing is turned on for the XXI routers.)
Q: Does the NetWatch database have information from csscplat devices?
A: Yes, as of December, 2005. When configuring switches, please be careful to preserve or put the MAC-learning IfTag on any uplinks when you configure them. This will ensure that NetWatch learns them, and also ensure that LAN stations aren't incorrectly displayed as being connected on the uplink ports. The catwatch utility is set up so that it will only learn MACs of csscplat switches if that device's config contains at least one "!M" tag; that is, it uses the presence of this tag to indicate that someone has disabled MAC learning on the uplink ports.
Q: What about wireless data?
A: Wireless mac address data is collected in a limited fashion. In NetWatch we collect 24 hours worth of data and the last location for each mac address. This data is only available for departmental SSID's. If you do not have access to this data, please submit a case with OpEng to be added as an admin or tech to the vlan in the wireless area that maps to your SSID.