Inter-subnet routing in EVPN MPLS Environment

In this article, we will review inter-subnet routing scenarios in EVPN environment. As we will discover, certain topologies might lead to sub-optimal traffic flows within EVPN network.

Our setup will be comprised of three PE and four CE devices as shown below:

Inter-subnet routing in EVPN Environment
Inter-subnet routing in EVPN Environment


Two Layer 3 subnets are used in this topology – 1.1.1.0/24 depicted as Green IRB and 1.1.2.0/24 depicted as Blue IRB. Both IRBs belong the same L3 VRF. Green and Blue IRBs are configured as L3 gateways for corresponding EVI interfaces.

PE2 device has local configuration for both Blue and Green EVI and IRB instances. PE5 has only Blue EVI/IRB and PE7 has Green EVI/IRB.

Logical Configuration

IP and MAC Assignments

Device IP MAC
CE2 1.1.1.2 00:0c:29:8a:79:e8
CE3 1.1.2.3 00:0c:29:de:e3:64
CE4 1.1.1.4 00:0c:29:82:c2:a9
CE7 1.1.2.7 00:0c:29:31:01:ed
PE2 IRB100 1.1.1.1 00:00:5e:01:00:00
PE2 IRB200 1.1.2.1 00:00:5e:02:00:00
PE5 IRB100 1.1.1.1 00:00:5e:01:00:00
PE7 IRB200 1.1.2.1 00:00:5e:02:00:00

PE2

Complete configuration on GitHub: PE2

CUSTOMER-A {
    instance-type virtual-switch;
    interface ge-0/0/1.0;
    interface ge-0/0/2.0;
    route-distinguisher 120.0.2.2:100;
    vrf-target target:100:100;
    protocols {
        evpn {
            extended-vlan-list [ 100 200 ];
            default-gateway do-not-advertise;
        }
    }
    bridge-domains {
        BD100 {
            vlan-id 100;
            routing-interface irb.100;
        }
        BD200 {
            vlan-id 200;
            routing-interface irb.200;
        }
    }
}

CUSTOMER-A-L3 {
    instance-type vrf;
    interface irb.100;
    interface irb.200;
    route-distinguisher 120.0.2.2:101;
    vrf-target target:100:101;
    vrf-table-label;
    protocols {
        evpn {
            ip-prefix-routes {
                advertise direct-nexthop;
            }
        }
    }
}

PE5

Complete configuration on GitHub: PE5

root@PE5> show configuration routing-instances
CUSTOMER-A {
    instance-type virtual-switch;
    interface ge-0/0/1.0;
    route-distinguisher 120.0.2.1:100;
    vrf-target target:100:100;
    protocols {
        evpn {
            extended-vlan-list 100;
            default-gateway do-not-advertise;
        }
    }
    bridge-domains {
        BD100 {
            vlan-id 100;
            routing-interface irb.100;
        }
    }
}

CUSTOMER-A-L3 {
    instance-type vrf;
    interface irb.100;
    route-distinguisher 120.0.2.1:101;
    vrf-target target:100:101;
    vrf-table-label;
    protocols {
        evpn {
            ip-prefix-routes {
                advertise direct-nexthop;
            }
        }
    }
}

PE7

Complete configuration on GitHub: PE7

root@PE7> show configuration routing-instances
CUSTOMER-A {
    instance-type virtual-switch;
    interface ge-0/0/0.0;
    route-distinguisher 120.0.2.7:100;
    vrf-target target:100:100;
    protocols {
        evpn {
            extended-vlan-list 200;
        }
    }
    bridge-domains {
        BD100 {
            vlan-id 200;
            routing-interface irb.200;
        }
    }
}
CUSTOMER-A-L3 {
    instance-type vrf;
    interface irb.200;
    route-distinguisher 120.0.2.7:101;
    vrf-target target:100:101;
    vrf-table-label;
    protocols {
        evpn {
            ip-prefix-routes {
                advertise direct-nexthop;
            }
        }
    }
}

Traffic flow scenarios

In this lab, we will review traffic flows originated and terminated on CE7. We have packet sniffer connected to PE7’s MPLS Network-facing interfaces, allowing us to capture packets with MPLS headers as they enter/leave PE device.

Scenario 1: CE7 <-> CE 3 traffic flow

In this scenario, we are reviewing traffic exchange between CE7 and CE3 devices. This is the simplest use case, as both CE3 and CE7 belong to the same L3 network. As such, no L3 routing is need and traffic is carried as Ethernet frames. This is not an inter-VLAN routing use case, but it’s included for completeness.

Scenario 1a: CE7 -> CE3

Inter-subnet routing in EVPN Environment - Scenario 1a
Inter-subnet routing in EVPN Environment – Scenario 1a

CE7:

CE7: $ traceroute 1.1.2.3
traceroute to 1.1.2.3 (1.1.2.3), 30 hops max, 60 byte packets
 1   1.1.2.3 (1.1.2.3)  59.903 ms  60.074 ms 59.603 ms  

CE7:$ arp -a | grep 1.1.2.3
1.1.2.3 at 00:0c:29:de:e3:64 [ether] on ens192

PE7:

root@PE7> show evpn database  extensive 
VLAN ID: 200, MAC address:: 00:0c:29:de:e3:64
Nexthop ID: 1048580
  Source: 120.0.2.2, Rank: 1, Status: Active                ß PE2
    MAC label: 300192
    Timestamp: Apr 29 07:12:07 (0x59047507)
    State: <Remote-To-Local-Adv-Done>
    IP address: 1.1.2.3
      L3 route: 1.1.2.3/32, L3 context: CUSTOMER-A-L3 (irb.200)

PE2:

root@PE2> show route label 300192
 
mpls.0: 21 destinations, 22 routes (21 active, 0 holddown, 0 hidden)
...
300192            

*[EVPN/7] 00:04:31, routing-instance CUSTOMER-A, 
route-type Ingress-MAC, vlan-id 100
                      to table CUSTOMER-A.evpn-mac.0

[EVPN/7] 00:01:52, routing-instance CUSTOMER-A, 
route-type Ingress-MAC, vlan-id 200
                      to table CUSTOMER-A.evpn-mac.0

Packet Capture:

MultiProtocol Label Switching Header, Label: 800202 (PE2), Exp: 0, S: 0, TTL: 255

MultiProtocol Label Switching Header, Label: 300192, Exp: 0, S: 1, TTL: 255

Ethernet II, Src: CE7(00:0c:29:31:01:ed), Dst: CE3(00:0c:29:de:e3:64)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 200
Internet Protocol Version 4, Src: 1.1.2.7, Dst: 1.1.2.3
Internet Control Message Protocol

Complete packet capture on GitHub: PCAP, Text, Text collapsed.

Scenario 1b: CE3 -> CE7

This scenario is a mirror image of the previous CE7 -> CE3 scenario. Detailed output is omitted for brevity.

Inter-subnet routing in EVPN Environment - Scenario 1b
Inter-subnet routing in EVPN Environment – Scenario 1b

Scenario 2: CE7 <-> CE 2 traffic flow

CE7 <-> CE2 traffic flow scenario represents an asymmetric configuration, where PE7 has no direct visibility into destination ‘Green’ Domain, while PE2 has the knowledge of PE7’s ‘Blue’ network.

As such, we might expect to see some asymmetry in traffic flow from and to CE7.

Scenario 2a: CE7 to CE2

Packet capture performed on PE7 demonstrates that CE7 to CE2 packet stream gets forwarded via PE5 device as shown below:

Inter-subnet routing in EVPN Environment - Scenario 2a
Inter-subnet routing in EVPN Environment – Scenario 2a

Clearly, this is sub-optimal scenario as packets are going via PE5 device, instead of more optimal PE7->PE2 path. Let’s review the routing information to understand the root cause.

CE7 resides on 1.1.2.0/24 subnet and has no direct 1.1.1.0/24 reachability. In order to send traffic to CE2, CE7 has to to rely on its default gateway 1.1.2.1. This gateway IP is configured on PE7’s ‘Blue’ IRB interface.

When PE7 gets an IP packet destined to 1.1.1.2, it performs an IP routing lookup to determine where to send this frame:

root@PE7> show route 1.1.1.2
CUSTOMER-A-L3.inet.0: 5 destinations, 7 routes 
1.1.1.0/24        
> to 120.0.3.32 via ge-0/0/1.570, Push 17, Push 800205(top)
> to 120.0.3.32 via ge-0/0/1.570, Push 17, Push 800202(top)

As you can see, specific 1.1.1.1/32 prefix is not being advertised by any of the devices. Instead, an aggregate 1.1.1.0/24 is being originated by both PE2 and PE5, as both devices have ‘Green’ IRB interfaces. PE7 has no way of determining which of two PEs has more optimal path to the specific 1.1.1.2/32 IP it is trying to reach and depending on the load-balancing policy, can pick either of them. In our case, PE5 device was selected leading to sub-optimal traffic flow.

When PE5 gets the packet, it performs a lookup in L3 VRF and determines that the packet should be redirected to PE2:

root@PE5> show route label 17
mpls.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)
17                 *[VPN/0] 22:49:56
> via lsi.1 (CUSTOMER-A-L3), Pop


root@PE5> show route 1.1.1.2 table CUSTOMER-A-L3.inet.0
CUSTOMER-A-L3.inet.0: 5 destinations, 7 routes 
1.1.1.2/32         *[EVPN/7] 13:33:21
> to 120.0.3.26 via ge-0/0/0.540, Push 300192, Push 800202(top)

It’s interesting to note that the information about 1.1.1.2/32 was derived from Type 2 advertisement received by PE5 from PE2:

root@PE5>  show route receive-protocol bgp 120.0.2.2
  2:120.0.2.2:100::100::00:0c:29:8a:79:e8::1.1.1.2/304 MAC/IP
*                         120.0.2.2                    100        I 

Scenario 2b: CE2 to CE7

Traffic in the reverse direction takes a different path. PE2 locally routes packet received via VLAN 100 to IRB100 (Green) and IRB200 (Blue).

Inter-subnet routing in EVPN Environment - Scenario 2b
Inter-subnet routing in EVPN Environment – Scenario 2b

When CE2 (1.1.1.2) needs to send packets to CE7 (1.1.2.7), it uses default gateway (1.1.1.1) that is configured on PE2. PE2, after receiving packets from CE2 does L3 lookup and forwards packets from IRB.100 to IRB.200. After that, PE2 can deliver packets to PE7’s EVI.

PE2:

root@PE2> show route 1.1.2.7 table CUSTOMER-A-L3.inet.0
CUSTOMER-A-L3.inet.0: 8 destinations, 10 routes 
1.1.2.7/32         *[EVPN/7] 15:48:24
> to 120.0.3.10 via ge-0/0/0.503, Push 299952, Push 800207(top)

PE7 pops MPLS label and delivers packets to CE7:

root@PE7> show route label 299952
mpls.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)

299952             *[EVPN/7] 15:53:02, routing-instance CUSTOMER-A, route-type Ingress-MAC, vlan-id 200
to table CUSTOMER-A.evpn-mac.0


root@PE7> show evpn database
Instance: CUSTOMER-A
VLAN  DomainId  MAC address        Active source                  Timestamp        IP address
100             00:0c:29:82:c2:a9  120.0.2.5                    1.1.1.4
100             00:0c:29:8a:79:e8  120.0.2.2                    1.1.1.2
200             00:00:5e:02:00:00  irb.200                      1.1.2.1
200             00:0c:29:31:01:ed  ge-0/0/0.0                   1.1.2.7
200             00:0c:29:de:e3:64  120.0.2.2                    1.1.2.3

Packet capture:

MultiProtocol Label Switching Header, Label: 299952, Exp: 0, S: 1, TTL: 253
Ethernet II, Src: irb200(00:00:5e:02:00:00), Dst: CE7 (00:0c:29:31:01:ed)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 200
Internet Protocol Version 4, Src: 1.1.1.2, Dst: 1.1.2.7
Internet Control Message Protocol

Complete packet capture on GitHub: PCAP, Text, Text collapsed.

Scenario 3: CE7 <-> CE 4 traffic flow

CE7 <-> CE4 traffic flow scenario represents symmetric scenario where PE devices have no visibility into each other’s EVI domains and must rely on IP routing in both directions.

Scenario 3a: CE7 to CE4

Similar to previously described CE7 to CE2 scenario, PE7 does IP lookup to determine 1.1.1.4’s reachability.

Inter-subnet routing in EVPN Environment - Scenario 3a
Inter-subnet routing in EVPN Environment – Scenario 3a

PE7:

root@PE7> show route 1.1.1.4

CUSTOMER-A-L3.inet.0: 5 destinations, 7 routes 

1.1.1.0/24         

*[EVPN/170] 1d 06:01:16
> to 120.0.3.32 via ge-0/0/1.570, Push 17, Push 800205(top)

[EVPN/170] 1d 06:01:16
> to 120.0.3.32 via ge-0/0/1.570, Push 17, Push 800202(top)

In our scenario, PE5 was selected as the egress LER, although PE2 could also have been selected.

When PE5 receives the packets, it forwards packets to CE4.

root@PE5> show route label 17 extensive

mpls.0: 21 destinations, 21 routes 
        *VPN    Preference: 0
                Next hop type: Router, Next hop index: 0
                Address: 0xb39e950
                Next-hop reference count: 1
                Next hop: via lsi.1 (CUSTOMER-A-L3), selected
                Label operation: Pop
                Load balance label: None;
                Label element ptr: 0xb39fc00
                Label parent element ptr: 0x0
                Label element references: 1
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 0x0
                State: <Active NotInstall Int Ext LsiL3>
                Age: 1d 8:47:59
                Validation State: unverified
                Task: RT
                AS path: I

root@PE5> show route 1.1.1.4/32 table CUSTOMER-A-L3.inet.0 extensive

CUSTOMER-A-L3.inet.0: 5 destinations, 7 routes 
1.1.1.4/32 (1 entry, 1 announced)
        *EVPN   Preference: 7
                Next hop type: Interface, Next hop index: 0
                Address: 0xb39f850
                Next-hop reference count: 2
                Next hop: via irb.100, selected
                State: <Active Int Ext>
                Age: 49
                Validation State: unverified
                Task: CUSTOMER-A-evpn
                Announcement bits (1): 0-Resolve tree 4
                AS path: I

root@PE5> show arp | match 1.1.1.4
00:0c:29:82:c2:a9 1.1.1.4         1.1.1.4                   irb.100 [ge-0/0/1.0

Please note that PE5 ge-0/0/1’s interface is configured as trunk:

root@PE5> show configuration interfaces ge-0/0/1
unit 0 {
    family bridge {
        interface-mode trunk;
        vlan-id-list 100;
    }
} 

Packet capture:

…
MultiProtocol Label Switching Header, Label: 800205, Exp: 0, S: 0, TTL: 255
MultiProtocol Label Switching Header, Label: 17, Exp: 0, S: 1, TTL: 255
Internet Protocol Version 4, Src: 1.1.2.7, Dst: 1.1.1.4
Internet Control Message Protocol

Complete packet capture on GitHub: PCAP, Text, Text collapsed.

Scenario 3b: CE4 to CE7

Inter-subnet routing in EVPN Environment - Scenario 3b
Inter-subnet routing in EVPN Environment – Scenario 3b

PE5:

root@PE5> show route 1.1.2.7


CUSTOMER-A-L3.inet.0: 5 destinations, 7 routes 
+ = Active Route, - = Last Active, * = Both

 
1.1.2.0/24         *[EVPN/170] 07:51:04
                    > to 120.0.3.26 via ge-0/0/0.540, Push 16, Push 800207(top)
                    [EVPN/170] 1d 06:20:45
                    > to 120.0.3.26 via ge-0/0/0.540, Push 17, Push 800202(top) 

 

PE7:

root@PE7> show route label 16 extensive
 
mpls.0: 21 destinations, 21 routes 
16  (1 entry, 0 announced)
        *VPN    Preference: 0
                Next hop type: Router, Next hop index: 0
                Address: 0xb39ea70
                Next-hop reference count: 1
                Next hop: via lsi.256 (CUSTOMER-A-L3), selected
                Label operation: Pop
                Load balance label: None;
                Label element ptr: 0xb39e9a0
                Label parent element ptr: 0x0
                Label element references: 1
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 0x0
                State: <Active NotInstall Int Ext LsiL3>
                Age: 1d 6:44:41
                Validation State: unverified
                Task: RT
                AS path: I

 

CUSTOMER-A-L3.inet.0: 5 destinations, 7 routes (5 active, 0 holddown, 0 hidden)

1.1.2.7/32 (1 entry, 1 announced)
        *EVPN   Preference: 7
                Next hop type: Interface, Next hop index: 0
                Address: 0xb39cd30
                Next-hop reference count: 2
                Next hop: via irb.200, selected
                State: <Active Int Ext>
                Age: 7:01
                Validation State: unverified
                Task: CUSTOMER-A-evpn
                Announcement bits (1): 0-Resolve tree 4
                AS path: I

root@PE7> show arp
00:0c:29:31:01:ed 1.1.2.7         1.1.2.7                   irb.200 [ge-0/0/0.0]   

Please note that ge-0/0/0 is configured as a trunk:

root@PE7> show configuration interfaces ge-0/0/0
unit 0 {
    family bridge {
        interface-mode access;
        vlan-id 200;
    }
}

Wireshark capture:


MultiProtocol Label Switching Header, Label: 16, Exp: 0, S: 1, TTL: 254
Internet Protocol Version 4, Src: 1.1.1.4, Dst: 1.1.2.7
Internet Control Message Protocol

Complete packet capture on GitHub: PCAP, Text, Text collapsed.

Complete Lab Repository with configuration files and packet captures.

Leave a Reply

Your email address will not be published. Required fields are marked *