MPLS Service Selection Guide
MPLS services can be used to extend your network transparently over the UW System Network MPLS core at layer 2 or layer 3. The first step of setting up a new MPLS service involves requirements gathering including information about intended use, reliability concerns, bandwidth needs and traffic priority compared to general internet traffic. With this information we can help guide you towards the appropriate MPLS solution. The following information is intended to help interested parties understand the various MPLS options available.
The UW System MPLS Network supports IP (layer 3) and ethernet (layer 2) VPNs. As of 2018/06/06 we have an example of each below type (pseudowire, redundant pseudowire, e-vpn, L3VPN) in production.
Stretching layer2 networks over long distances is generally recommended against due to concerns of LAN segmentation. Layer3 VPNs, where workable, are the recommended solution.
There may be licensing limits to the number of routing-instances (L3VPN, EVPN), but not pseudowire circuits. We can likely solve this with $$$.
Layer3 VPNs: Layer3 VPNs are multipoint and support IPv4 and IPv6 unicast. If you need IP multicast for your application, a layer 3 VPN may still be a possibility however this is not something we've done in production and we may choose to steer you towards another VPN solution.
When using Layer3 VPNs, you configure connector adjacencies (BGP multihomed preferred, non multihomed static supported) with the MPLS core, advertising routes as needed. In general the MPLS core operator need not be involved with the routes being advertised other than generally putting a limit on the number of prefixes allowed to be advertised from each node.
As we track RIB changes we gain an good amount of visibility of network health and are in a good position to help if something goes bad.
Layer2 VPNs: We offer pseudowire (Martini) and Ethernet VPN (e-vpn) services. We are not currently offering l2vpn (Kompella) or VPLS services.
Pseudowire, also known as l2circuit or "Martini" based L2VPNs, are simple to configure. They can best be through of as a virtual circuit across the MPLS network. Bits that ingress a logical interface generally egress as-is on a different logical interface. A logical interface could be a physical interface, an LACP bundle or a set of vlans on either a physical interface or LACP bundle. Tunneling can include protocols such as LLDP, LACP, BPDUs.
Not much can go wrong, so it's unlikely we'll need to be involved in troubleshooting, but at the same time all we can tell you is if the virtual pipe is up or down.
[FIXME: show examples. show examples of uwm or marshfield nursing]
MPLS core operators like l2circuits as they are robust and scalable as no MAC learning occurs in the MPLS core. In most scenarios you do not need to contact the MPLS core provider if you add/change vlans. The downside to l2circuit is that it is not a multipoint solution. While many psuedowires we configure are point to point, pseudowire offers a redundancy feature that allows you to have three nodes in a 'Y' pattern. The redundant leg of the pseudowire is pre-signaled in the MPLS core but can be thought of as being 'blocked' like in a spanning tree network. The block is removed if the main pseudowire fails. Failure reasons would include a loss of reachability in the MPLS core or a link down event on either endpoint. Internal LAN reachability problems do -not- cause a failover. Restoration to the main pseudowire is a toggleable option.
[FIXME: show examples. UW Stout]
Ethernet VPN, also known as E-VPN, provides true multipoint ethernet extension over the WAN. But with this power comes several caveats.
In comparison to pseudowire, E-VPN configurations are complex and require the service provider to coordinate much more closely with the connector. The MPLS core provider must have a detailed understanding of the LAN topology to help configure the E-VPN service correctly. The MPLS core provider must be consulted when vlans are added or removed. The MPLS core is involved in MAC learning and is responsible for loop prevention and broadcast/multicast replication as needed.
Each mac address requires a forwarding entry in each PE where the routing-instance exists, which has a cost. Specific to our current network, MX104 as licensed supports 32K entries, underlying hardware will reach 1M but from a convergence perspective that is highly unrealistic on the MX104.
We use scripts and templates to help maintain configuration correctness, however, more vendor codepath is being exercised so there is more room for error.
As of 2023 we are able to track mac address changes inside an E-VPN with exaBGP, so we may be able to provide some information if/when something goes bad.
[FIXME: show examples. UWW]
FAQs:
Q: Does the MPLS core provide encryption?
A: We are not currently providing network encryption via the MPLS core and we do not have the hardware capability to do so. We strongly recommend end-to-end encryption to ensure data integrity and privacy between endpoints including on the local LAN.
Q: How does the MPLS core handle bandwidth contention?
A: The UW System Network core offers four standard QoS queues; best-effort, expedited-forwarding, assured-forwarding and network-control. Each MPLS service can be configured differently. The MPLS backbone imposes MPLS EXP bits for QoS on the MPLS frames. This topic will be discussed when a new service is brought online.
- The best-effort queue is split into two packet loss priorities (PLP), low and high, providing us the ability to offer less-than-best-effort service. This would allow interfaces to be saturated absent any congestion.
- The expedited-forwarding queue is split into two packet loss priorities (PLP). Up to 20% of line rate will be scheduled before best-effort but this queue cannot starve the best-effort queue.
- The assured-forwarding queue is scheduled before the expedited-forwarding queue. It is best used for IP voice and IP video. The % of line rate varies by interface speed, but this queue cannot starve the expedited-forwarding queue. MPLS services are currently not allowed to admit traffic into this queue.
- The network-control queue is scheduled before all other queues but cannot starve the assured-forwarding queue. MPLS services are not allowed to admit traffic into this queue; it is reserved explicitly for MPLS core management.
Q: Can I have an optical circuit instead of using MPLS?
A: In nearly all cases a shared network design is a better choice. In terms of resiliency, as of 2016/11/28, several UW System Network connectors do not have optical network redundancy into the UW System core and rely on third party circuits (WIN, BadgerNet) for redundant connectivity. MPLS will work over nearly all vendor circuits.
In our current design, optical circuits are added in 100G optical carrier group (OCG) increments with a single OCG support ten 10G waves. Since optical transport is a limited resource, we must balance the use of waves for shared applications vs dedicated resources.
Specifically, MPLS services can be configured to use any available path for forwarding, for example, taking advantage of tertiary IP paths. In contrast, optical resources are strictly provisioned for a single use. Redundant optical services requires strict reservation that cannot be used for other services. By adding capacity to the IP/MPLS backbone we also grow capacity for general internet traffic and the ability to handle unexpected surges of data (sometimes unwanted, such as a DoS attack.