MPLS operations, administration, and maintenance (OAM) is a set of tools that helps to troubleshoot MPLS networks. One of the MPLS OAM tools is MPLS ping. In this article, I will describe why normal IP ping is insufficient for verifying Label Switched Path (LSP), and how MPLS ping solves that problem.
Diagram
- Every router has a Loopback0 configured with IP address X.X.X.X/32, where X is the router number, i.e R2 (2.2.2.2/32).
- Transit links are configured with IP addresses 10.X.Y.0/24, where X and Y are adjacent router numbers, i.e. the link between R1 and R2 is 10.1.2.0/24
- In SP 65100 network OSPF is configured as IGP.
- OSPF cost of the link P-R4 (towards PE-XR5) is artificially increased, so the preferred path from PE-XR1 perspective is through P-R2 -> PR3 - > PE-XR5.
- LDP is enabled on the SP network, except the link on P-R4 (towards P-R2) and both links on P-R4.
- Customer A has sites (AS 65001 and 65002) connected via MPLS SP network (AS 65100). Service provider is offering L3VPN service and the ultimate goal is to provide reachability between sites - R6 (6.6.6.6) should be able to ping R7 (7.7.7.7).
Configuration
PE-XR1
hostname PE-XR1
!
vrf CUSTOMER
address-family ipv4 unicast
import route-target
1:1
!
export route-target
1:1
!
!
!
interface Loopback0
ipv4 address 1.1.1.1 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
shutdown
!
interface GigabitEthernet0/0/0/0
ipv4 address 10.1.2.1 255.255.255.0
!
interface GigabitEthernet0/0/0/1
ipv4 address 10.1.4.1 255.255.255.0
!
interface GigabitEthernet0/0/0/2
vrf CUSTOMER
ipv4 address 10.1.6.1 255.255.255.0
!
route-policy PASS
pass
end-policy
!
router ospf 1
area 0
interface Loopback0
!
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
!
router bgp 65100
address-family vpnv4 unicast
!
neighbor 5.5.5.5
remote-as 65100
update-source Loopback0
address-family vpnv4 unicast
!
!
vrf CUSTOMER
rd 1:1
address-family ipv4 unicast
!
neighbor 10.1.6.6
remote-as 65001
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
!
!
!
!
mpls oam
!
mpls ldp
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
end
P-R2
hostname P-R2
!
ip cef
!
interface Loopback0
ip address 2.2.2.2 255.255.255.255
!
interface FastEthernet0/0
ip address 10.1.2.2 255.255.255.0
duplex full
mpls ip
!
interface FastEthernet1/0
ip address 10.2.3.2 255.255.255.0
duplex full
mpls ip
!
router ospf 1
network 0.0.0.0 255.255.255.255 area 0
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
P-R3
hostname P-R3
!
ip cef
!
interface Loopback0
ip address 3.3.3.3 255.255.255.255
!
interface FastEthernet0/0
ip address 10.3.5.3 255.255.255.0
duplex full
mpls ip
!
interface FastEthernet1/0
ip address 10.2.3.3 255.255.255.0
duplex full
! MPLS not enabled
!
router ospf 1
network 0.0.0.0 255.255.255.255 area 0
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
P-R4
hostname P-R4
!
ip cef
!
interface Loopback0
ip address 4.4.4.4 255.255.255.255
!
interface FastEthernet0/0
ip address 10.4.5.4 255.255.255.0
ip ospf cost 1000
duplex full
!
interface FastEthernet1/0
ip address 10.1.4.4 255.255.255.0
duplex full
!
router ospf 1
network 0.0.0.0 255.255.255.255 area 0
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
PE-XR5
hostname PE-XR5
!
vrf CUSTOMER
address-family ipv4 unicast
import route-target
1:1
!
export route-target
1:1
!
!
!
interface Loopback0
ipv4 address 5.5.5.5 255.255.255.255
!
interface MgmtEth0/0/CPU0/0
shutdown
!
interface GigabitEthernet0/0/0/0
ipv4 address 10.3.5.5 255.255.255.0
!
interface GigabitEthernet0/0/0/1
ipv4 address 10.4.5.5 255.255.255.0
!
interface GigabitEthernet0/0/0/2
vrf CUSTOMER
ipv4 address 10.5.7.5 255.255.255.0
!
route-policy PASS
pass
end-policy
!
router ospf 1
area 0
interface Loopback0
!
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
!
!
router bgp 65100
address-family vpnv4 unicast
!
neighbor 1.1.1.1
remote-as 65100
update-source Loopback0
address-family vpnv4 unicast
!
!
vrf CUSTOMER
rd 1:1
address-family ipv4 unicast
!
neighbor 10.5.7.7
remote-as 65002
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
!
!
!
!
mpls oam
!
mpls ldp
interface GigabitEthernet0/0/0/0
!
interface GigabitEthernet0/0/0/1
!
CE-R6
hostname CE-R6
!
ip cef
!
interface Loopback0
ip address 6.6.6.6 255.255.255.255
!
interface FastEthernet0/0
ip address 10.1.6.6 255.255.255.0
duplex full
!
router bgp 65001
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.1.6.1 remote-as 65100
!
address-family ipv4
network 6.6.6.6 mask 255.255.255.255
network 10.1.6.0 mask 255.255.255.0
neighbor 10.1.6.1 activate
exit-address-family
CE-R7
hostname CE-R7
!
ip cef
!
interface Loopback0
ip address 7.7.7.7 255.255.255.255
!
interface FastEthernet0/0
ip address 10.5.7.7 255.255.255.0
duplex full
!
router bgp 65002
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.5.7.5 remote-as 65100
!
address-family ipv4
network 7.7.7.7 mask 255.255.255.255
network 10.5.7.0 mask 255.255.255.0
neighbor 10.5.7.5 activate
exit-address-family
Verification
In this case service provider is utilizing MPLS, where LDP is used in conjunction with IGP and the goal is to provide the transport mechanism for customer - in this case, L3VPN. Label Switched Path formed between PE routers must be end-to-end. In the shown network that is not true - LDP is not enabled on the link on P-R3 (towards P-R2) and P-R4 is not running LDP completely.
As you may guess at this point, there is no connectivity between Customer sites.
CE-R6#ping 7.7.7.7 source 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6
.....
Success rate is 0 percent (0/5)
CE-R6#show ip route bgp | begin Gateway
Gateway of last resort is not set
7.0.0.0/32 is subnetted, 1 subnets
B 7.7.7.7 [20/0] via 10.1.6.1, 00:07:15
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
B 10.5.7.0/24 [20/0] via 10.1.6.1, 00:07:15
Clients are getting proper routes (output from CE-R7 not shown for brevity), so it’s reasonable to assume that the problem is not related to the BGP itself. The First come to mind is to test connectivity between PE-XR1 and PE-XR5. Is there connectivity between them?
RP/0/0/CPU0:PE-XR1#ping 5.5.5.5 source 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 5.5.5.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 29/35/49 ms
The main point of this article is to show that ICMP ping can be successful with a broken or even no LSP. Based on the diagram, there is no LDP session between P-R2 and P-R3, which results in a broken LSP. As seen in the output above ping indicates that there is no problem as the echo request makes it to the destination (PE-XR5) and the echo reply makes it back to the source (PE-XR1). Between the two routers (P-R2 and P-R3) where the LDP session is broken, the packets are no longer labeled. The ping indicates falsely that everything is okay, when in reality the LSP is broken.
P-R2#show mpls ldp discovery
Local LDP Identifier:
2.2.2.2:0
Discovery Sources:
Interfaces:
FastEthernet0/0 (ldp): xmit/recv
LDP Id: 1.1.1.1:0
! Interface towards P-R3
FastEthernet1/0 (ldp): xmit
P-R2#show mpls ldp neighbor | i Peer LDP Ident
Peer LDP Ident: 1.1.1.1:0; Local LDP Ident 2.2.2.2:0
P-R2#show mpls forwarding-table 5.5.5.5
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
20 No Label 5.5.5.5/32 9612 Fa1/0 10.2.3.3
When troubleshooting broken LSP, checking every hop can be a laborious and time-consuming task. Some causes of the LSP failure may be:
- Broken LDP adjacency
- MPLS not enabled (globally or per interface)
- Software or hardware corruption
In summary - ICMP ping verifies if there is connectivity in the IP level, but it does not verify whether the LSP is broken. That’s why MPLS ping was invented.
Important note: in order to MPLS ping work on the IOS XR platform, it must be explicitly enabled by issuing the command: mpls oam.
While traditional ping command utilizes ICMP protocol, MPLS ping has a new packet format and uses UDP port 3503 for destination and source port chosen by a sender.
Echo request uses data path LSP for the specified FEC (in our case it will be IPv4 prefix - 5.5.5.5/32). The destination IP address within echo request packet is 127.0.0.0/8 and IP TTL is set to 1. Because of that, a router that is the destination or router in the middle that detects the failed path will generate an echo reply.
RP/0/0/CPU0:PE-XR1#ping mpls ipv4 5.5.5.5/32
Sending 5, 100-byte MPLS Echos to 5.5.5.5/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
BBBBB
Success rate is 0 percent (0/5)
MPLS ping shows that there is an unlabeled output interface, which results in a broken LSP. Let’s fix it by enabling LDP on the link on P-R3.
P-R3(config-if)# mpls ip
%LDP-5-NBRCHG: LDP Neighbor 2.2.2.2:0 (2) is UP
LSP is now continuous from the PE-XR1 perspective and MPLS ping confirms that.
RP/0/0/CPU0:PE-XR1#ping mpls ipv4 5.5.5.5/32
Sending 5, 100-byte MPLS Echos to 5.5.5.5/32,
timeout is 2 seconds, send interval is 0 msec:
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 10/22/40 ms
Let’s try to investigate packet header in order to better understand MPLS ping.
At this point, MPLS ping confirms LSP path, you may be thinking that R6 can ping R7.
CE-R6#ping 7.7.7.7 source 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6
.....
Success rate is 0 percent (0/5)
But there is still no connectivity. Why? Remember LSPs are unidirectional and MPLS ping verifies LSP in only one direction from PE-XR1 to PE-XR5. LSP between PE-XR5 and PE-XR1 is still broken because LDP was not enabled on P-R4.
While MPLS echo request takes MPLS path, MPLS reply may take MPLS or an IP path. In this case echo reply generated by PE-XR5 was sent through the bottom path (via P-R4, which was IP path, not MPLS), because of the lower OSPF cost to reach PE-XR1.
Let’s enable LDP on P-R4.
P-R4(config)#router ospf 1
P-R4(config-router)#mpls ldp autoconfig
LDP-5-NBRCHG: LDP Neighbor 1.1.1.1:0 (1) is UP
%LDP-5-NBRCHG: LDP Neighbor 5.5.5.5:0 (2) is UP
MPLS ping issued on PE-XR5 confirms that LSP is formed in the way back.
RP/0/0/CPU0:PE-XR5#ping mpls ipv4 1.1.1.1/32
Sending 5, 100-byte MPLS Echos to 1.1.1.1/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/34/50 ms
Now, we successfully provide connectivity between Customer sites.
CE-R6#ping 7.7.7.7 source 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/53/72 ms