E-VPN

E-VPN basic terminology

BUM: Broadcast, unknown unicast, and multicast traffic. Essentially multi-destination traffic.

DF: Designated Forwarder: The EVPN PE responsible for forwarding BUM traffic from the core to the CE. Once a set of multi-homed PE peers have discovered each other, a PE is elected as the Designated Forwarder (DF) for the ES (Ethernet Segment). The DF is responsible for transmitting BUM traffic from the core to the CE. The non-DF, or Backup Forwarder, PE drops BUM traffic received from the core destined to the CE.

DF essentially does the job of spanning tree.

ES: Ethernet Segment. The Ethernet link(s) between a CE device and one or more PE devices. In a multi-homed topology the set of links between the CE and PEs is considered a single “Ethernet Segment.” Each ES is assigned an identifier.

ESI: Ethernet Segment Identifier. A 10 octet value with range from 0x00 to 0xFFFFFFFFFFFFFFFFFFFF which represents the ES. An ESI must be set to a network-wide unique, non-reserved viii value when a CE device is multi-homed to two or more PEs. This triggers the advertisement of an MP-BGP Ethernet Segment route by each of the multi-homed PEs that allows them to automatically discover each other. For a single homed CE the reserved ESI value 0 is used. The ESI value of “all FFs” is also reserved.

Ethernet Tag Identifier: Identifies the broadcast domain in an EVPN instance. For our purposes the broadcast domain is a VLAN and the Ethernet Tag Identifier is the VLAN ID.

EVI: EVPN Instance, defined on PEs to create the EVPN service.

IP VPN - a Layer 3 VPN service implemented using BGP/MPLS IP VPNs (RFC 4364)

MAC-VRF: MAC address virtual routing and forwarding table. This is the Layer 2 forwarding table on a PE for an EVI.

MP2MP: Multipoint to Multipoint.

P2MP: Point to Multipoint.

PMSI: Provider multicast service interface. A logical interface in a PE that is used to deliver multicast packets from a CE to remote PEs in the same VPN, destined to CEs.

Determining the DF for an ES

DF election is performed ESI per EVI. This facilitates load balancing of BUM traffic amongst PEs, a feature known as Service Carving. See: https://tools.ietf.org/html/rfc7432#section-8.5

@r-mx104-lab-ac-re0> show evpn instance EVPN-2 esi 00:00:07:00:00:00:00:00:00:00 extensive
Instance: EVPN-2
Route Distinguisher: 143.235.32.106:700
Per-instance MAC route label: 299776
MAC database status                Local Remote
    Total MAC addresses:                 0       3
    Default gateway MAC addresses:       0       0
Number of local interfaces: 1 (1 up)
    Interface name ESI                            Mode             Status
    ae1.200         00:00:07:00:00:00:00:00:00:00 single-active    Up
Number of IRB interfaces: 0 (0 up)
Number of bridge domains: 3
    VLAN ID Intfs / up    Mode             MAC sync IM route label
    200          1   1     Extended         Enabled   308256
    201          1   1     Extended         Enabled   308272
    202          1   1     Extended         Enabled   308288
Number of neighbors: 2
    143.235.32.38
      Received routes
        MAC address advertisement:              0
        MAC+IP address advertisement:           0
        Inclusive multicast:                    3
        Ethernet auto-discovery:                2
    143.235.32.113
      Received routes
        MAC address advertisement:              3
        MAC+IP address advertisement:           0
        Inclusive multicast:                    3
        Ethernet auto-discovery:                0
Number of ethernet segments: 1
    ESI: 00:00:07:00:00:00:00:00:00:00
      Status: Resolved by NH 1048577
      Local interface: ae1.200, Status: Up/Blocking
      Number of remote PEs connected: 1
        Remote PE        MAC label Aliasing label Mode
        143.235.32.38    306720     306720          single-active
      Designated forwarder: 143.235.32.38
      Backup forwarder: 143.235.32.106
      Advertised MAC label: 308240
      Advertised aliasing label: 308240
      Advertised split horizon label: 0

Other useful commands

Another way to check e-vpn forwarding state on a CE link

@r-mx104-lab-ac-re0# run show interfaces ae1.200 detail | match EVPN
Protocol bridge, MTU: 1522, Generation: 222, Route table: 4, Mesh Group: __all_ces__, EVPN multi-homed status: Blocking

Looking at an E-VPN bridge table

@r-mx104-lab-ac-re0> show bridge mac-table instance EVPN-2

MAC flags       (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
    O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

Routing instance : EVPN-2
Bridging domain : V200, VLAN : 200
   MAC                 MAC      Logical          NH     RTR
   address             flags    interface        Index ID
   00:11:20:3e:3e:81   DC                        1048594 1048594
   00:11:20:3e:3e:c2   DC                        1048594 1048594
   00:24:97:32:a7:42   DC                        1048577 1048577

MAC flags       (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
    O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

Routing instance : EVPN-2
Bridging domain : V201, VLAN : 201
   MAC                 MAC      Logical          NH     RTR
   address             flags    interface        Index ID
   00:11:20:3e:3e:81   DC                        1048594 1048594

MAC flags       (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
    O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

Routing instance : EVPN-2
Bridging domain : V202, VLAN : 202
   MAC                 MAC      Logical          NH     RTR
   address             flags    interface        Index ID
   00:11:20:3e:3e:81   DC                        1048594 1048594

@r-mx104-lab-ac-re0> show evpn database instance EVPN-2 extensive
Instance: EVPN-2

VLAN ID: 200, MAC address: 00:11:20:3e:3e:81
Source: 143.235.32.113, Rank: 1, Status: Active
    Timestamp: Sep 11 10:18:23 (0x55f2f0bf)
    State: <Local-Adv-Allowed Local-Adv-Done>

VLAN ID: 200, MAC address: 00:11:20:3e:3e:c2
Source: 143.235.32.113, Rank: 1, Status: Active
    Timestamp: Sep 11 15:29:13 (0x55f33999)
    State: <Local-Adv-Allowed Local-Adv-Done>

VLAN ID: 200, MAC address: 00:24:97:32:a7:42
Source: 00:00:07:00:00:00:00:00:00:00, Rank: 1, Status: Active
    Remote origin: 143.235.32.38
    Timestamp: Sep 11 15:29:13 (0x55f33999)
    State: <Local-Adv-Allowed Local-Adv-Done>

VLAN ID: 201, MAC address: 00:11:20:3e:3e:81
Source: 143.235.32.113, Rank: 1, Status: Active
    Timestamp: Sep 11 10:18:25 (0x55f2f0c1)
    State: <Local-Adv-Allowed Local-Adv-Done>

VLAN ID: 202, MAC address: 00:11:20:3e:3e:81
Source: 143.235.32.113, Rank: 1, Status: Active
    Timestamp: Sep 11 10:18:25 (0x55f2f0c1)
    State: <Local-Adv-Allowed Local-Adv-Done>

Looking at routing per mac address

@r-mx104-lab-ac-re0> show route table EVPN-2.evpn.0 evpn-mac-address 00:24:97:32:a7:42

EVPN-2.evpn.0: 16 destinations, 16 routes (16 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

1:143.235.32.38:0::0700000000000000::FFFF:FFFF/304

*[BGP/170] 3d 03:23:26, localpref 100, from 143.235.32.112

AS path: I, validation-state: unverified

> to 143.235.33.217 via ae0.3477

to 143.235.33.144 via xe-0/3/0.3475, Push 299888

1:143.235.32.38:700::0700000000000000::0/304

*[BGP/170] 3d 03:23:26, localpref 100, from 143.235.32.112

AS path: I, validation-state: unverified

> to 143.235.33.217 via ae0.3477

to 143.235.33.144 via xe-0/3/0.3475, Push 299888

2:143.235.32.38:700::200::00:24:97:32:a7:42/304

*[BGP/170] 00:03:47, localpref 100, from 143.235.32.112

AS path: I, validation-state: unverified

> to 143.235.33.217 via ae0.3477

to 143.235.33.144 via xe-0/3/0.3475, Push 299888

Route tables

bgp.evpn.0:

Contains all EVPN related routes carried in BGP. To decode, see https://tools.ietf.org/html/rfc7432#section-7.

In the routes, the first integer is the E-VPN NLRI Route type.

Ethernet Auto-Discovery (A-D) route: In a multi-homed configuration, auto-Discovery per ESI and per EVI is needed.

Route Formats

[@$cms ~]$ egrep -v "(AS path|Validation|Age|State|Indirect|Next-hop reference|Address: 0x|Local AS:|Source:|Protocol next hop:|Originator ID:|Cluster list:|Localpref:|Router ID:)" tmp2
@r-mx104-lab-ac-re0> show route table bgp.evpn.0 detail

bgp.evpn.0: 14 destinations, 14 routes (14 active, 0 holddown, 0 hidden)
1:143.235.32.38:0::0700000000000000::FFFF:FFFF/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:0
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700 esi-label:single-active (label 0)
                Import Accepted
                Secondary Tables: EVPN-2.evpn.0

The Auto-Discovery per ESI route is used for fast convergence and for preventing the looping of BUM traffic. It is advertised by both multi-homed PEs connected to the ES.

1:143.235.32.38:700::0700000000000000::0/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:700
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Route Label: 306720
                Secondary Tables: EVPN-2.evpn.0

When a PE router detects a new MAC address on its EVI access interface, it adds the address to its appropriate local Layer 2 forwarding table, or MAC-VRF. The PE then transmits a MAC Advertisement route using MP-BGP to all remote PEs.

The inclusion of the ESI in the MAC Advertisement route is critical for implementing aliasing, or load balancing. Multi-homed PEs advertise their connectivity to a common ESI by transmitting Auto-Discovery routes to all remote PEs. When a given remote PE subsequently learns of a MAC address from that ESI, it knows that the destination is reachable via the set of multi-homed PEs. The PE can then load balance traffic to the multiple PEs connected to the common ES.

2:143.235.32.38:700::200::00:24:97:32:a7:42/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:700
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Route Label: 306720
                ESI: 00:00:07:00:00:00:00:00:00:00
                Secondary Tables: EVPN-2.evpn.0

2:143.235.32.113:700::200::00:11:20:3e:3e:81/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.113:700
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Route Label: 299776
                ESI: 00:00:00:00:00:00:00:00:00:00
                Secondary Tables: EVPN-2.evpn.0

2:143.235.32.113:700::200::00:11:20:3e:3e:c2/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.113:700
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Route Label: 299776
                ESI: 00:00:00:00:00:00:00:00:00:00
                Secondary Tables: EVPN-2.evpn.0

2:143.235.32.113:700::201::00:11:20:3e:3e:81/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.113:700
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Route Label: 299776
                ESI: 00:00:00:00:00:00:00:00:00:00
                Secondary Tables: EVPN-2.evpn.0

2:143.235.32.113:700::202::00:11:20:3e:3e:81/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.113:700
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Route Label: 299776
                ESI: 00:00:00:00:00:00:00:00:00:00
                Secondary Tables: EVPN-2.evpn.0

Each EVPN PE advertises an Inclusive Multicast (IM) route to enable forwarding of BUM traffic. The PMSI Tunnel Attribute is the same attribute that is used in Next Generation BGP Multicast VPNs (https://tools.ietf.org/html/rfc6513). It includes the Tunnel Type that indicates the multicast technology to be used in the core network to forward BUM traffic. In the case of type ingress replication, when a PE receives a BUM packet from a CE device, it makes a copy of the packet corresponding to each of the remote PEs. It then encapsulates each packet with the appropriate MPLS labels before forwarding the packets. In order to simplify forwarding in the core while independently scaling the number of EVIs at the edge, the initial implementation of EVPN in Junos supports ingress replication. The trade-off in this case is the Chapter 3: Verification 49 additional processing by the ingress PE to duplicate and transmit the BUM packets. However, note that on Junos platforms the PFE, not the routing engine, performs efficient internal multicast replication using binary trees.

3:143.235.32.38:700::200::143.235.32.38/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:700
                PMSI: Flags 0x0: Label 306736: Type INGRESS-REPLICATION 143.235.32.38
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Secondary Tables: EVPN-2.evpn.0

3:143.235.32.38:700::201::143.235.32.38/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:700
                PMSI: Flags 0x0: Label 306752: Type INGRESS-REPLICATION 143.235.32.38
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Secondary Tables: EVPN-2.evpn.0

3:143.235.32.38:700::202::143.235.32.38/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:700
                PMSI: Flags 0x0: Label 306768: Type INGRESS-REPLICATION 143.235.32.38
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Secondary Tables: EVPN-2.evpn.0

3:143.235.32.113:700::200::143.235.32.113/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.113:700
                PMSI: Flags 0x0: Label 306464: Type INGRESS-REPLICATION 143.235.32.113
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Secondary Tables: EVPN-2.evpn.0

3:143.235.32.113:700::201::143.235.32.113/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.113:700
                PMSI: Flags 0x0: Label 306480: Type INGRESS-REPLICATION 143.235.32.113
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Secondary Tables: EVPN-2.evpn.0

3:143.235.32.113:700::202::143.235.32.113/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.113:700
                PMSI: Flags 0x0: Label 306496: Type INGRESS-REPLICATION 143.235.32.113
                Task: BGP_65010.143.235.32.112+179
                Communities: target:65010:700
                Import Accepted
                Secondary Tables: EVPN-2.evpn.0

4:143.235.32.38:0::0700000000000000:143.235.32.38/304 (1 entry, 0 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:0
                Task: BGP_65010.143.235.32.112+179
                Communities: es-import-target:7-0-0-0-0-0
                Import Accepted
                Secondary Tables: __default_evpn__.evpn.0

__default_evpn__.evpn.0: used for carrying MP-BGP information about ethernet segment routes. This is only applicable for PE that have non default ESI [ie, multihomed CE]. All EVPN NLRI Type 4 routes are also stored in the secondary __default_evpn__.evpn.0 table since they do not contain a Route Target community that corresponds to any specific EVI.

[the below output has been truncated for readability]
@r-mx104-lab-ac-re0# run show route table __default_evpn__.evpn.0 detail

Note that in below, the EVPN ES (ethernet segment) route, NLRI type 4, are isolated via show command by matching on ^4:$ip.

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
1:143.235.32.106:0::0700000000000000::FFFF:FFFF/304 (1 entry, 1 announced)
        *EVPN   Preference: 170
                Task: __default_evpn__-evpn
                Announcement bits (1): 1-BGP_RT_Background
                Communities: target:65010:700 esi-label:single-active (label 0)

4:143.235.32.38:0::0700000000000000:143.235.32.38/304 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 143.235.32.38:0
                Task: BGP_65010.143.235.32.112+179
                Announcement bits (1): 0-__default_evpn__-evpn
                Communities: es-import-target:7-0-0-0-0-0
                Import Accepted
                Primary Routing Table bgp.evpn.0

4:143.235.32.106:0::0700000000000000:143.235.32.106/304 (1 entry, 1 announced)
        *EVPN   Preference: 170
                Task: __default_evpn__-evpn
                Announcement bits (1): 1-BGP_RT_Background
                Communities: es-import-target:7-0-0-0-0-0

$instance.evpn.0 is like bgp.evpn.0 but is routing-instance specific

P has bgp.evpn.0 NLRI

@r-mx2010-lab-re0> show bgp summary
143.235.32.113        65010      31806      32336       0       0 1w2d 20:43:49 Establ
...
bgp.evpn.0: 6/6/6/0

@r-mx2010-lab-re0# run show route summary
Autonomous system number: 65010
Router ID: 143.235.32.111
...
...
inet.3: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
              Static:      1 routes,      1 active
                 LDP:      3 routes,      3 active

mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
                MPLS:      6 routes,      6 active
                 LDP:      6 routes,      6 active

bgp.evpn.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
                 BGP:     17 routes,     17 active

PE has many other NLRI as it has a route target in the E-VPN VRF

@r-mx104-lab-ac-2-re0> show bgp summary
...
143.235.32.112        65010      32350      31809       0       1 1w2d 20:46:18 Establ
bgp.evpn.0: 9/9/9/0
EVPN-2.evpn.0: 9/9/9/0 <------------
__default_evpn__.evpn.0: 0/0/0/0

@r-mx104-lab-ac-re0# run show route summary
Autonomous system number: 65010
Router ID: 143.235.32.106
...
...

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
                 LDP:      3 routes,      3 active

mpls.0: 25 destinations, 25 routes (25 active, 0 holddown, 0 hidden)
                MPLS:      6 routes,      6 active
                 CCC:      2 routes,      2 active
                 LDP:      5 routes,      5 active
                EVPN:     12 routes,     12 active

bgp.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
                 BGP:     13 routes,     13 active

EVPN-2.evpn.0: 15 destinations, 15 routes (15 active, 0 holddown, 0 hidden)
                 BGP:     12 routes,     12 active
                EVPN:      3 routes,      3 active

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
                 BGP:      1 routes,      1 active
                EVPN:      2 routes,      2 active

E-VPN with default gateways on each PE

Once the MAC/IP binding of a given host is learned by the PE, it then transmits another MAC Advertisement route which contains both the MAC and IP addresses. This process is also known as Host MAC/IP Synchronization. Note that an IRB interface for the EVPN VLAN must be configured in order for the PE to transmit MAC/IP Advertisement routes, which was NOT part of the initial E-VPN lab.

A PE also advertises a MAC/IP Advertisement route containing the IP and MAC address of the locally configured IRB interface along with the Default Gateway Extended Community. The Default Gateway Extended Community signals to the receiving PE that it must route traffic on behalf of the advertising PE. This process is also referred to as Default Gateway Synchronization. The MAC/IP Advertisements are essential to the integration of Layer 3 routing with Layer 2 EVPNs,

EVPN lessons learned

* Like other MPLS VPNs you can use LDP or RSVP; in our case as of 2015/09/11 we are mostly using LDP + LDP LFA for quick failover.

* ESI is non default if a CE has path to more than one PE even if the path is through the CE's switched network, not directly connected. In this case single-active must be used.

* MAC Mobility

When a PE learns of a source MAC address via its local EVI access interface, it transmits a MAC Advertisement route to all other PEs. However, if the MAC address moves to another Ethernet Segment, such that it is now reachable via another PE, the original PE receives an MP-BGP announcement from the new PE. The original PE updates its forwarding table with the new reachability information, which then triggers the withdrawal of its previously advertised MAC Advertisement route.

The MAC Mobility Extended Community includes a sequence number that increments with each MAC move. This is used to by PEs to ensure that the MAC Advertisements are processed correctly. It can also be used to detect MAC flapping. For example, if a PE detects that a number of MAC moves within a given time period exceeds a configured threshold it can alert the network operator and stop sending MAC Advertisement

routes. This community will be supported in an upcoming release of Junos.

to explore/understand later

On the data center PEs the protocol type BGP routes, learned via the IP VPN, are redundant. Therefore, to reduce overhead you can optionally discard these routes by applying policies to the IP VPN instance’s VRF. An example of this is configured in the Chapter 2 section: Configuration Services, where a unique community is added to the advertised IP VPN routes such that they are discarded when received by remote data center PEs.

Basic config example

PE:

@r-mx104-lab-ac-re0# show interfaces ae1
apply-groups-except ethernet-standards;
description "s-lab-4 Port-channel1 uwplatteville switch 2 simulator";
enable;
per-unit-scheduler;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
esi {           <---------- omit this if single homed
    00:00:00:00:00:00:00:00:07:00;
    single-active;
}
aggregated-ether-options {
    link-speed 1g;
    lacp {
        active;
    }
}
unit 200 {
    description "s-lab-4 Port-channel1 uwplatteville switch 2 simulator";
    family bridge {
        interface-mode trunk;
        vlan-id-list [ 200 201 202 ];
    }
}

@r-mx104-lab-ac-re0# show routing-instances EVPN-2
instance-type virtual-switch;
interface ae1.200;
route-distinguisher 143.235.32.106:700;
vrf-target target:65010:700;
protocols {
    evpn {
        extended-vlan-list 200-202;
        default-gateway advertise;
    }
}
bridge-domains {
    V200 {
        vlan-id 200;
    }
    V201 {
        vlan-id 201;
    }
    V202 {
        vlan-id 202;
    }
}

@r-mx104-lab-ac-re0# show protocols bgp group iBGP-reflector-client family evpn
signaling;

P:

set protocols bgp group iBGP-reflector family evpn signaling
set policy-options policy-statement select-iBGP-reflector-routes term bgp-evpn from protocol bgp
set policy-options policy-statement select-iBGP-reflector-routes term bgp-evpn from rib bgp.evpn.0
set policy-options policy-statement select-iBGP-reflector-routes term bgp-evpn then next policy

Keywords:

E-VPN basic terminology

Doc ID:

56133

Owned by:

Michael H. in UW System Network

Created:

2015-09-11

Updated:

2024-04-23

Sites:

University of Wisconsin System Network

5 0 Comment Suggest new doc