1. Moving less connections (21 connections in rack A3 vs 140 connections in rack A2).
2. Will allow us to swap out A2 and A3 2 post racks for 4 post racks (once the next gen equipment migration occurs A2 will be empty).
3. The high density connection network equipment will be closest to the fiber racks.
4. Easier to move the power.

Dennis and Neil will be putting in a request to see if we can get the room to the east of b3a at 432 N. Murray. This would get us a 10 ft x 31 ft room to expand into.

1. Only connect the -9 and -10 to the ASR 9922's.
2. Connect the 6500's to the ASR 9922's and move routing to the 9922's.

We discussed where to physically store the licenses for the 7k's and 9k's, but did not come up with a decision. Greg proposed moving vlan 494 behind one of the new firewalls and we agreed this made sense.

• \NS_Shared_Docs\Projects\Next Gen Network
• Al let us know that Housing has 24 of their 33 aggregation buildings have singlemode.
• Ryan T. let us know that Cisco is coming out with a new smaller 6500 with a new sup that could be a good radial instead of the 4500X. The new 6500 will be announced this summer. Ryan also let know that he and Al Kluender have created ASR 9k training. The State of Minnesota will be the first customer to take it. We will be the second and it is scheduled for:
  • When: Wednesday, April 03, 2013, 9:00 AM to Friday, April 05, 2013, 5:00 PM. (UTC-06:00) Central Time (US & Canada)
  • Where: Madison (maybe Cisco office on the eastside or at Union South).

  The format that was described was we would be given a task to configure and we would look up how to do it via the internet. We would be given a show command to let us know if what is configured is correct. We would be given the directions/training on how to configure. This does not use up any training credits. All engineers, OpEng, Mark Tinberg and Michael Hare are interested in attending (11 people in all). Consensus is to have the time changed to 8:00a.m. to 4:00 p.m.

1. Intro to 9K a. Hardware b. Software c. Packet walks
2. Intro to XR
3. XR software architecture
4. XR Operations Tools
5. IOS XR Routing 6. IOS XR RPL 7. MPLS 8. Multicast 9. QOS 10. L2 11. Software Upgrade

• Greg and David: CSSC-1 and CSSC-2 Bill and Will: 432 N. Murray Jeff and John: Animal-1 and Animal-2

We will also configure the ASR 9k as it relates to the Nexus 7k pair that we are configuring.

• Jeff/John : Animal Science 9922
• Greg/David: CSSC 9922
• Bill/Will: Techlab 9922
• Mark/Scott: Techlab 9006

The 7k boards have been installed at CSSC and 432 N. Murray. We will install the Animal Science ones when they are racked up. We need to have terminal server and network connections run at each site. We will have to install terminal server and management switch at Animal Science room 226.

• "Here are the action items from this mornings Next-Gen Backbone meeting, if I have missed something please let me know.
  
  Thanks,
  Bill
  • Tom follow-up:
    • IPv6 virtual address ??
    • HSRP authentication corrected in 4.3.1 - already answered still not available in 4.3.1
  • UW follow-up:
    • BFD
      • keep horseshoe between 9Ks or P2P between 9K & 7K?
      • Use BFD on physical P2P vs Logical HSRP - test response time and try to determine which combination is most appropriate
    • OSPF
      • better redistribution than just "redistribute connected" best practices
    • HSRP - test response time of using BFD vs HSRP timers"

• Cutover service by service or everything for a building all at once? No decision made.
• Consensus to use 3850's as radial/access switch for buildings that have 4 or less equipment closets ( 1 MDF and 3 IDF's).
• Consensus is to have 2 PoE 48 port 3650's per access switch stack if we go with the 3650's.
• We need to draw up diagrams to help with decisions/discussions for how to implement the cutover. Scott has a diagram that we did not have time to go over.
• Mark Karls and Scott let us know that the camera network is targeted to be completed by October 2014.

• Te1/1/1   VSL 1
  Te1/1/2   VSL 2
  Te1/1/3   Uplink
  Te1/1/4   TR 1
  Te1/1/5   TR 2
  Te1/1/6   TR 3
  Te1/1/7   TR 4
  Te1/1/8   TR 5
  Te1/1/9   TR 6
  Te1/1/10  TR 7
  Te1/1/11  TR 8
  Te1/1/12  TR 9
  Te1/1/13  Uplink - reserved
  Te1/1/14  Dual-active detection
  Te1/1/15  VSL 3
  Te1/1/16  VSL 4
  
  
  Te2/1/1   VSL 1
  Te2/1/2   VSL 2
  Te2/1/3   Uplink
  Te2/1/4   TR 1
  Te2/1/5   TR 2
  Te2/1/6   TR 3
  Te2/1/7   TR 4
  Te2/1/8   TR 5
  Te2/1/9   TR 6
  Te2/1/10  TR 7
  Te2/1/11  TR 8
  Te2/1/12  TR 9
  Te2/1/13  Uplink - reserved
  Te2/1/14  Dual-active detection
  Te2/1/15  VSL 3
  Te2/1/16  VSL 4
  

• NOTE: BAN ASAs will have to be connected to the s-bldg-mdf-2-access stack due to lack of 100 Mbps support on the 4500x.
• Scott has created Visio masters in the DoIT-Base-Stencil stencil located in the NS_Shared_Docs/Networks/Radials/0000_EXAMPLE folder.

• USB drives for 7k's. Consensus is to make sure they stay in the 7k's in the log flash port. Format the usb drives. Need to make sure they are in log flash port. Will let us know that he is working with Jim Shaft to move the ones at 432 to the log flash port. Consensus is we need an auditing tool too make sure the usb drives are in the correct port and formatted. Janet will add this to the daily cms report and put it on the Network Services LAN Team Programmer ToDo List.
• Ordering Transition Networks serial extender parts: Consensus is to get 1 fully populated chassis and 1 Cisco 2901 per supernode and reserve space for future expansion.
• Consensus for ordering cwdm equipment is: 2 chassis per supernode with fully populated MUX modules. Ordering takes 4-6 weeks. Spare on all of the colors being ordered, totally populate 3 colors to start with 66% 1 Gig and 33% 10 Gig. 2 spares per color. Using the first two colors for 1 Gig (1471 and 1491) and the last color for 10 Gig (1611). This will allow us to grow from both ends towards the middle.
• State of Wisconsin looking to borrow some spare 7k cards for 3 weeks per Greg. We have 1 gig boards in our 7k’s that we probably will not need now. Consensus is yes on the 1 gig cards.

• We discussed extending routing down to the 7k’s vs ASR 9k’s. One of the questions is if the number of OSPF areas is supportable and best practices to do this on the 9k’s? Tom Glennon said yes, it would work but is this widely supported and deployed. We discussed doing something that looks closer to other deployments vs being a corner case. We discussed doing an area per building due to limitation on code running on the radials. We discussed building a separate vDC for routing on the 7k’s. We discussed minimizing the number of routes getting to the radial/access level. We discussed using mpls with the radial being the CE and 7k’s being the PE. How many customer networks per radial? What is the limit? Greg let us know that there are 663 firewalled networks on campus on our firewall service. The current plan is to migrate Cisco firewalls to the Nexus 7k’s. We were considering moving firewalling to the ASR 9k’s. We need to address arp, mac address and multicast limits. We discussed using a service box for cross campus layer 2 separate from the dedicated wireless Nexus 6k’s. Tom Glennon let us know that he will go back to the Nexus BU and follow up on our questions about gotcha’s/limitations. Tom took notes on what to follow up with. We will continue this discussion on Wednesday, Dennis will get a room reserved for this. We discussed using FEXes connected to the M2 cards for 1 Gig connections to the radials.

• Dennis let us know that Kevin Schmidtke invited him Friday to attend a meeting to discuss monitoring services. Not all network performance events are detected by FIDO. Jim Winkle has created some latency and ping tests that did detect the service issue when we had the ASIC issue with the Cisco 6500’s. There is a desire to figure out how to incorporate Jim’s tests into FIDO. Possibly using Solar Winds to monitor applications will also be discussed. End user experience monitoring is also desirable.

• WiscNic global vlan cleanup is needed for 220 global vlans that are also area scoped. This will be added as a JIRA item. Logan is working on the tool for auditing this.
• We went over the Techlab layout depicted on the whiteboard in Mark Tinberg’s cube. What code base to use for radials (LAN versus ipbase) will be determined by testing in the Techlab.

• “Scott, Dennis and Team,
  
  Sorry for the delay in getting back to you on this question. I’ve been out with the stomach flu most of this week. We are still working to get a Nexus 7k engineer for an on-site visit( waiting for the TME manager to provide us with a replacement resource for Ron Fuller). Also, I have another engineer mocking up the scenario or having an L2 orphan port on the same physical trunk as other L3 interfaces. So far it looks like it should be supported but we would like take a few more days to confirm this configuration. Again, sorry for the delay in getting a definitive answer on this question.
  
  Thanks,
  
  Tom Glennon”
• David and John have the 9k configurations in the Techlab almost ready. Mark Karls and Jeff Robertson are almost done with the 7k configurations in the Techlab.

• The PCI builder is on hold because we need input from Mark Karls. The radial builder tool needs to link to Scott’s NextGenNetBuilder tool to get the Nexus interface configuration. Use the NextGenNetBuilder without the firewall option to build point to points. Configuring the transit using the Next Gen builder from the BAN cgi tool. BAS (vlan 4072) is one network behind the ASA, note that this is not the transit for the ASA. BAS will eventually become a vrf. The ASA5585’s have been moved to the Nexus 7k’s per Greg, so CoE can have their context moved there.

• Scott let us know that 10 gig portal channels at 432nm and CSSC had traffic stop, rebuilding it corrects the problem. Creating the ports first then the port channel is the recommended order, but this still created the problem with the WEI configuration. Tom Glennon will be going over MPLS options this Wednesday.
• Mark Karls and Will let us know that PCI has urpf problem that causes dhcp to not work. We went over the milestone tracker project NextGen Radial Migrations and assigned people to buildings.

• We discussed trunking our vlan in the Techlab to test the new Nexus 7010 code. We could dual connect our network in the Techlab to make it easy to flip back and forth. We could do HSRP on 4 routers instead of 2. The lab Nexus 7k’s would route our traffic when we go into test mode. We could also change the weight on the ASR 9k’s so it prefers the Techlab Nexus 7k’s during the testing. There is a risk of layer 2 loops. Can a loop in one vdc impact another vdc? Can we setup vm’s in the Techlab for testing instead of using our network? This would be beta testing in addition to proof of concept testing. There will have to be a freeze on Techlab activities when doing the beta testing. Would this replace deploying on one third of campus? We will think about this more and we will have to discuss this with everyone on the 144.92.67.0/24 network (e.g. WAN team, Field Services, Programmers…).

• “FIDO currently does monitor power usage " (POE), see NS-3143. The data is available in GNMIS with dataset "=PoEAllocatedPower". A report is being generated and is available here.
• archive: https://stats.net.wisc.edu/cgi-bin/rrd_reports.cgi?report=poe_usage
• direct link to most recent report: https://stats.net.wisc.edu/reports/current/poe_usage_daily.html
• FIDO alarms for PoeAllocatedPower are not being created at this point awaiting (to the best of my knowledge) a bit of QA from Mark Karls.
• I see NS-3437 has been opened. That request is much more than what NS-3143 does, which is per switch allocated tracking.”
• There are currently 4 locations that are over the threshold of 90% (memun and chadhall). A goal is to not have unnecessary/false FIDO alerts. Consensus is to change the threshold to 100% when we have a switch stack that is fully populated with the biggest power supplies. When there are FIDO alerts bring them to the engineering meeting. The report is showing higher than what the show power inline (e.g. 91% on the report versus 86% using show power inline).
• The switch stack at Mem Union should have another power supply installed in it. Mark Karls and David will work on the QA portion so that FIDO alerting can be done. Mark Karls will cut a ticket to have Field Services install a power supply at Mem Union.
• Consensus is to configure port priority for edge ports for power shedding to keep AP’s and cameras up versus keeping other devices like VOIP phones up. Consensus is to set VOIP phones to low priority and AP’s and cameras to high. Consensus is this should not be a edgeconf setting. Consensus is to have the default port priority as low and cameras/AP’s as high. Use show lldp neighbors detail to generate a report for setting the priority to high. Mark Karls is going to take this request to Mark Tinberg and Tom Christie. Going forward we should make sure that the setting is high(power inline port priority high) for AP’s and cameras.

• Dustin said the optics were over heated. ASR at CSSC filter intake screen was plugged. The sfp’s were replaced since they were causing failures (e.g. port suspends).
• Dennis let us know that the Cisco clock component failure replacement parts are here. Bill has worked on replacing parts in the Techlab.

• 9600 cons:
  • no front to back airflow
  • 12 to 18 months for new sup at the earliest
  • feature set does not currently meet needs for supernodes
  • requires new tooling
  • no deep buffers
  • new code and architecture (e.g. high probability unknown bugs that will be part of new code and hardware)
  • would be a one off (e.g. additional sparing)
• 7706 pluses
  • fast deployment
  • front to back airflow
  • can use current tooling
  • it scales for routing
  • we know it is going to work
  • better economy of sparing (e.g. wireless, -9, -10 are 7706’s)
  • better TCAM scaling
• We discussed if we should use 7700 chassis and cards versus getting M3 cards for 7010 chassis for the supernodes. Pricing comparison between using 7706 and 7010 options for true 100 Gig core. We could use 7710’s to get more slots since we will be using all slots with the 7706 chassis right from the start.
• 7010 solution:
  • Cost 
    • M3 card N7k 1/10 48 port card : $85,000
    • M3 card N7k 40 gig 24 port : $85,000
    • M202’s two 100 Gig ports: $85,000
    • Optics: $3,000
  • Considerations
    • we have to remove the M1 cards
    • last about 3 years
    • we cannot go to newer code with M2 cards
    • M2 has flow limitations of around 12 Gig per flow which is probably an issue going forward
    • have to upgrade code to use newer cards (e.g. M3 cards)
    • we could use M2’s to get to 100 Gig for core connections
• 7710 chassis solution:
  • Cost 
    • M3 card N7k 1/10 48 port card : $65,000
    • M3 card N7k 40 gig 24 port : $75,000
    • M202’s twelve port 100 Gig: $115,000
    • Optics: $800
    • Sup cards: $40,000 (need 2)
    • Fab 3: $15,000 (need 6)
    • Chassis: $15,000
    • Power supplies: (4 are needed)
  • Considerations
    • there are chassis bundles
    • we can get rid of the Cisco 3232 switches
    • last longer about 5 years
    • we could go to 400 Gig
    • true netconf for better programmability
    • VXLAN all ports
    • no safe harbor release

• We discussed creating a site to site vpn service. Dan let us know that SMPH does want a site to site vpn service. Chris Spencer asked Dan to come up with pricing for MUFN connections. UWHealth wants us to put in as little as possible equipment and to not use more fiber. We discussed using client based vpn for sites that have only a couple of users. Is terminating the vpn connection to campus O.K. or does the encryption need to go to the building? Looks like Palo Alto 820 for $5,000 per and there would be two. The Palo Alto 820 would terminate back to the campus Palo Alto core. Palo Alto will have SDWAN in 9.1 code per Scott.

• Access switch:
  • C9300-48UN, cold spare 8, 1 hot spare
• Radial switch:
  • C9300X-12Y, cold spare 2, 1 hot spare
  • C9300X-24Y, cold spare 2, 1 hot spare
  • C9300-48UN, cold spare 2, 1 hot spare
• Modules:
  • C9300-NM-8X, cold spare 2
  • C9300X-NM-2C, cold spare 2
  • C9300X-NM-2Y, cold spare 2
• Dennis has ordered some C9300-48UN switches with modules. The order should arrive in December. We can stack together 8 switches. Dennis thinks these are in the CBS system. We need cards for the Nexus core routers. Mark Karls is going to come with the list of buildings with 3750X's. Consensus is to deploy 9300 switches with 10 Gig uplinks for the 3750X building refresh. John will investigate current 1 gig connections to see if they can go to 10 gig. We are connecting to the existing 7010's for now. Dennis is going to talk to Neil about the power needs now that we have the switch models. We want an electrician from FP&M to help figure out what to do in the closets. Scott put the power requirements on the Visio diagram that he created. 2200W/240V = 9A or 2200W/120V = 20A times the number of switches. Scott will add the new switch models to the Visio network template with the modules. John will update the hot and cold sparing KB with the new models. Work on switch configuration tools for the new models has started. Dennis thinks we have one of each switch in the Techlab.