LDP and RSVP-TE are both signaling protocols used for establishing LSP in the MPLS networks. The latter one provides traffic engineering capability and in this article, I will try to set basic MPLS-TE tunnels on IOS XR devices to show you a problem, you can run into. A label switched path (LSP) is a sequence of LSR that switch a labeled packet through an MPLS network. As long LSP is contiguous (end-to-end between PE routers) there should be no problem in forwarding packets. Typically in the service provider core network, LDP will be enabled and RSVP-TE will run on top of it, but in this scenario, LDP is not enabled.
Diagram
- Every router has a Loopback0 configured with IP address X.X.X.X/32, where X is the router number, i.e R2 (2.2.2.2/32).
- Transit links are configured with IP addresses 10.X.Y.0/24, where X and Y are adjacent router numbers, i.e. the link between R1 and R2 is 10.1.2.0/24.
- In SP 65100 network OSPF is configured as IGP with RSVP-TE enabled on every core link.
- RSVP-TE tunnels are set between PE routers (IOS XR) with an explicit path configured that packets will be routed through a longer path: XR1-> R2 -> R3 -> XR5. Note that tunnels are unidirectional, so two tunnels are configured (PE-XR1 to PE-XR5 and PE-XR5 to PE-XR1).
- Customer A has sites (AS 65001 and 65002) connected via MPLS SP network (AS 65100). Service provider is offering L3VPN service and the ultimate goal is to provide reachability between sites - R6 (6.6.6.6) should be able to ping R7 (7.7.7.7).
Configuration
PE-XR1
hostname PE-XR1
explicit-path name XR1_R2_R3_XR5
index 10 next-address strict ipv4 unicast 2.2.2.2
index 20 next-address strict ipv4 unicast 3.3.3.3
index 30 next-address strict ipv4 unicast 5.5.5.5
!
vrf CUSTOMER
address-family ipv4 unicast
import route-target
1:1
!
export route-target
1:1
!
!
!
interface Loopback0
ipv4 address 1.1.1.1 255.255.255.255
!
interface tunnel-te0
ipv4 unnumbered Loopback0
autoroute announce
!
destination 5.5.5.5
path-option 1 explicit name XR1_R2_R3_XR5
!
interface MgmtEth0/0/CPU0/0
shutdown
!
interface GigabitEthernet0/0/0/0
ipv4 address 10.1.2.1 255.255.255.0
!
interface GigabitEthernet0/0/0/1
ipv4 address 10.1.4.1 255.255.255.0
!
interface GigabitEthernet0/0/0/2
vrf CUSTOMER
ipv4 address 10.1.6.1 255.255.255.0
!
route-policy PASS
pass
end-policy
!
router ospf 1
area 0
mpls traffic-eng
interface Loopback0
!
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
mpls traffic-eng router-id Loopback0
!
router bgp 65100
address-family vpnv4 unicast
!
neighbor 5.5.5.5
remote-as 65100
update-source Loopback0
address-family vpnv4 unicast
!
!
vrf CUSTOMER
rd 1:1
address-family ipv4 unicast
!
neighbor 10.1.6.6
remote-as 65001
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
!
!
!
!
mpls oam
!
rsvp
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
mpls traffic-eng
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
P-R2
hostname P-R2
!
ip cef
!
mpls traffic-eng tunnels
!
interface Loopback0
ip address 2.2.2.2 255.255.255.255
!
interface FastEthernet0/0
ip address 10.1.2.2 255.255.255.0
duplex full
mpls traffic-eng tunnels
ip rsvp bandwidth
!
interface FastEthernet1/0
ip address 10.2.3.2 255.255.255.0
duplex full
mpls traffic-eng tunnels
ip rsvp bandwidth
!
router ospf 1
network 0.0.0.0 255.255.255.255 area 0
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
P-R3
hostname P-R3
!
ip cef
!
mpls traffic-eng tunnels
!
interface Loopback0
ip address 3.3.3.3 255.255.255.255
!
interface FastEthernet0/0
ip address 10.3.5.3 255.255.255.0
duplex full
mpls traffic-eng tunnels
ip rsvp bandwidth
!
interface FastEthernet1/0
ip address 10.2.3.3 255.255.255.0
duplex full
mpls traffic-eng tunnels
ip rsvp bandwidth
!
router ospf 1
network 0.0.0.0 255.255.255.255 area 0
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
P-R4
hostname P-R4
!
ip cef
!
mpls traffic-eng tunnels
!
interface Loopback0
ip address 4.4.4.4 255.255.255.255
!
interface FastEthernet0/0
ip address 10.4.5.4 255.255.255.0
duplex full
mpls traffic-eng tunnels
ip rsvp bandwidth
!
interface FastEthernet1/0
ip address 10.1.4.4 255.255.255.0
duplex full
mpls traffic-eng tunnels
ip rsvp bandwidth
!
router ospf 1
network 0.0.0.0 255.255.255.255 area 0
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
PE-XR5
hostname PE-XR5
explicit-path name XR5_R3_R2_XR1
index 10 next-address strict ipv4 unicast 3.3.3.3
index 20 next-address strict ipv4 unicast 2.2.2.2
index 30 next-address strict ipv4 unicast 1.1.1.1
!
vrf CUSTOMER
address-family ipv4 unicast
import route-target
1:1
!
export route-target
1:1
!
!
!
interface Loopback0
ipv4 address 5.5.5.5 255.255.255.255
!
interface tunnel-te0
ipv4 unnumbered Loopback0
autoroute announce
!
destination 1.1.1.1
path-option 1 explicit name XR5_R3_R2_XR1
path-option 10 dynamic
!
interface MgmtEth0/0/CPU0/0
shutdown
!
interface GigabitEthernet0/0/0/0
ipv4 address 10.3.5.5 255.255.255.0
!
interface GigabitEthernet0/0/0/1
ipv4 address 10.4.5.5 255.255.255.0
!
interface GigabitEthernet0/0/0/2
vrf CUSTOMER
ipv4 address 10.5.7.5 255.255.255.0
!
route-policy PASS
pass
end-policy
!
router ospf 1
area 0
mpls traffic-eng
interface Loopback0
!
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
mpls traffic-eng router-id Loopback0
!
router bgp 65100
address-family vpnv4 unicast
!
neighbor 1.1.1.1
remote-as 65100
update-source Loopback0
address-family vpnv4 unicast
!
!
vrf CUSTOMER
rd 1:1
address-family ipv4 unicast
!
neighbor 10.5.7.7
remote-as 65002
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
!
!
!
!
mpls oam
!
rsvp
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
mpls traffic-eng
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
CE-R6
hostname CE-R6
!
ip cef
!
interface Loopback6
ip address 6.6.6.6 255.255.255.255
!
interface FastEthernet0/0
ip address 10.1.6.6 255.255.255.0
duplex full
!
router bgp 65001
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.1.6.1 remote-as 65100
!
address-family ipv4
network 6.6.6.6 mask 255.255.255.255
network 10.1.6.0 mask 255.255.255.0
neighbor 10.1.6.1 activate
exit-address-family
CE-R7
hostname CE-R7
!
ip cef
!
interface Loopback0
ip address 7.7.7.7 255.255.255.255
!
interface FastEthernet0/0
ip address 10.5.7.7 255.255.255.0
duplex full
!
router bgp 65002
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.5.7.5 remote-as 65100
!
address-family ipv4
network 7.7.7.7 mask 255.255.255.255
network 10.5.7.0 mask 255.255.255.0
neighbor 10.5.7.5 activate
exit-address-family
Verification and Troubleshooting
As stated in the beginning the goal is to provide reachability for Customer, so that R6 can ping R7.
CE-R6#ping 7.7.7.7 source 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6
.....
Success rate is 0 percent (0/5)
Looks like it there is some issue in the network that needs to be fixed. First, let’s check if the control plane is working, specifically if R6 and R7 have routes to each other. It’s easy to make mistake, simply by mistyping route-target numbers and then routes are not properly inserted into the correct VRF.
CE-R6#show bgp ipv4 unicast
BGP table version is 7, local router ID is 6.6.6.6
Network Next Hop Metric LocPrf Weight Path
*> 6.6.6.6/32 0.0.0.0 0 32768 i
*> 7.7.7.7/32 10.1.6.1 0 65100 65002 i
*> 10.1.6.0/24 0.0.0.0 0 32768 i
*> 10.5.7.0/24 10.1.6.1 0 65100 65002 i
CE-R6#show ip route bgp | begin Gateway
Gateway of last resort is not set
7.0.0.0/32 is subnetted, 1 subnets
B 7.7.7.7 [20/0] via 10.1.6.1, 04:29:54
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
B 10.5.7.0/24 [20/0] via 10.1.6.1, 04:29:54
CE-R7#show bgp ipv4 unicast
BGP table version is 7, local router ID is 7.7.7.7
Network Next Hop Metric LocPrf Weight Path
*> 6.6.6.6/32 10.5.7.5 0 65100 65001 i
*> 7.7.7.7/32 0.0.0.0 0 32768 i
*> 10.1.6.0/24 10.5.7.5 0 65100 65001 i
*> 10.5.7.0/24 0.0.0.0 0 32768 i
CE-R7#show ip route bgp | begin Gateway
Gateway of last resort is not set
6.0.0.0/32 is subnetted, 1 subnets
B 6.6.6.6 [20/0] via 10.5.7.5, 00:04:38
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
B 10.1.6.0/24 [20/0] via 10.5.7.5, 04:30:07
That’s not the case here. R6 and R7 are using BGP as PE-CE protocol and both routers have received routes and then those routes were properly inserted into the RIB.
The next step in troubleshooting is to check the service provider network. As mentioned earlier to provide L3VPN service in the SP network must be set LSP between PE routers - end-to-end. Maybe mpls traffic-engineering or rsvp was not enabled on some interface. All routers should have the same Traffic Engineering Database (TED), which contains topology information. The quickest way to verify that is to issue the command with filters that make output more
RP/0/0/CPU0:PE-XR1#show mpls traffic-eng topology | include "MPLS TE Id|Frag Id"
IGP Id: 1.1.1.1, MPLS TE Id: 1.1.1.1 Router Node (OSPF 1 area 0)
Frag Id:3, Intf Address:10.1.2.1, Intf Id:0
Frag Id:4, Intf Address:10.1.4.1, Intf Id:0
IGP Id: 2.2.2.2, MPLS TE Id: 2.2.2.2 Router Node (OSPF 1 area 0)
Frag Id:1, Intf Address:10.1.2.2, Intf Id:0
Frag Id:2, Intf Address:10.2.3.2, Intf Id:0
IGP Id: 3.3.3.3, MPLS TE Id: 3.3.3.3 Router Node (OSPF 1 area 0)
Frag Id:1, Intf Address:10.3.5.3, Intf Id:0
Frag Id:2, Intf Address:10.2.3.3, Intf Id:0
IGP Id: 4.4.4.4, MPLS TE Id: 4.4.4.4 Router Node (OSPF 1 area 0)
Frag Id:1, Intf Address:10.4.5.4, Intf Id:0
Frag Id:2, Intf Address:10.1.4.4, Intf Id:0
IGP Id: 5.5.5.5, MPLS TE Id: 5.5.5.5 Router Node (OSPF 1 area 0)
Frag Id:4, Intf Address:10.3.5.5, Intf Id:0
Frag Id:5, Intf Address:10.4.5.5, Intf Id:0
The output confirms that mpls traffic-eng is enabled on all interfaces in the core network. Based on configuration PE-XR1 should use explicit path R2->R3->XR5 (same is true for the opposite direction, but I will omit to show output for brevity).
RP/0/0/CPU0:PE-XR1#show mpls traffic-eng tunnels role head
Name: tunnel-te0 Destination: 5.5.5.5 Ifhandle:0x70
Signalled-Name: PE-XR1_t0
Status:
Admin: up Oper: up Path: valid Signalling: connected
path option 1, type explicit XR1_R2_R3_XR5 (Basis for Setup, path weight 3)
G-PID: 0x0800 (derived from egress interface properties)
Bandwidth Requested: 0 kbps CT0
Creation Time: Tue Sep 14 12:02:53 2021 (04:22:05 ago)
Config Parameters:
Bandwidth: 0 kbps (CT0) Priority: 7 7 Affinity: 0x0/0xffff
Metric Type: TE (global)
Path Selection:
Tiebreaker: Min-fill (default)
Hop-limit: disabled
Cost-limit: disabled
Path-invalidation timeout: 10000 msec (default), Action: Tear (default)
AutoRoute: enabled LockDown: disabled Policy class: not set
Forward class: 0 (default)
Forwarding-Adjacency: disabled
Autoroute Destinations: 0
Loadshare: 0 equal loadshares
Auto-bw: disabled
Fast Reroute: Disabled, Protection Desired: None
Path Protection: Not Enabled
BFD Fast Detection: Disabled
Reoptimization after affinity failure: Enabled
Soft Preemption: Disabled
History:
Tunnel has been up for: 04:21:03 (since Tue Sep 14 12:03:55 UTC 2021)
Current LSP:
Uptime: 04:21:03 (since Tue Sep 14 12:03:55 UTC 2021)
Reopt. LSP:
Last Failure:
LSP not signalled, identical to the [CURRENT] LSP
Date/Time: Tue Sep 14 12:56:35 UTC 2021 [03:28:23 ago]
Path info (OSPF 1 area 0):
Node hop count: 3
Hop0: 10.1.2.2
Hop1: 10.2.3.2
Hop2: 10.2.3.3
Hop3: 10.3.5.3
Hop4: 10.3.5.5
Hop5: 5.5.5.5
Displayed 1 (of 1) heads, 0 (of 0) midpoints, 0 (of 1) tails
Displayed 1 up, 0 down, 0 recovering, 0 recovered heads
This is lengthy output, but the most important information is that path was successfully calculated and signaled. Final verification is LSP is to end-to-end is to traceroute from XR1 to XR5 (and in the opposite direction).
RP/0/0/CPU0:PE-XR1#traceroute 5.5.5.5 source 1.1.1.1
Tue Sep 14 16:29:58.806 UTC
Type escape sequence to abort.
Tracing the route to 5.5.5.5
1 10.1.2.2 [MPLS: Label 16 Exp 0] 39 msec 59 msec 49 msec
2 10.2.3.3 [MPLS: Label 16 Exp 0] 39 msec 39 msec 39 msec
3 10.3.5.5 39 msec * 49 msec
RP/0/0/CPU0:PE-XR1#ping mpls traffic-eng Tunnel-te 0
Tue Sep 14 16:30:21.175 UTC
Sending 5, 100-byte MPLS Echos to tunnel-te0,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/38/60 m
Both traceroute and ping mpls shows that LSP is end-to-end between PE routers. Note: In order mpls ping to work mpls oam was configured on both XR routers.
TE Tunnels are formed and labels are allocated, but clients still can’t ping each other. A solution to fix this problem is kinda obscure - you can spend hours verifying configuration and still have this problem. IOS XR routers (PE) are missing command: mpls ldp
It’s not so obvious because LDP is not running in the core. Also, there is no need to enable any interfaces under mpls ldp section. In the current state, there is no connectivity.
RP/0/0/CPU0:PE-XR1#show cef vrf CUSTOMER 7.7.7.7/32
7.7.7.7/32, version 7, internal 0x5000001 0x0 (ptr 0xa13f2284) [1], 0x0 (0x0), 0x208 (0xa1495140)
Updated Sep 14 11:53:48.541
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via 5.5.5.5/32, 0 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0xa0d10294 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
unresolved
labels imposed {24000}
Next-hop is unresolved. After configuring mpls ldp command the output changes to:
RP/0/0/CPU0:PE-XR1#show cef vrf CUSTOMER 7.7.7.7/32
7.7.7.7/32, version 7, internal 0x5000001 0x0 (ptr 0xa13f2284) [1], 0x0 (0x0), 0x208 (0xa1495140)
Updated Sep 14 11:53:48.542
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via 5.5.5.5/32, 3 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0xa14fd75c 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 5.5.5.5/32 via 24008/0/21
next hop 5.5.5.5/32 tt0 labels imposed {ImplNull 24000}
And most importantly customer sites can ping each other.
CE-R6#ping 7.7.7.7 source 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/102/176 ms