The ones and twos of the to- and through-

Picture the scene…

We have four sites. Each site has a VPN to each of the other sites, forming a mesh, with default routes pointing to the vIOS router.

Traffic from each of the PCs to the other PCs works fine. All is good, right?

Here are the pings from PC3:

VPCS> ping 172.16.1.10

172.16.1.10 icmp_seq=1 timeout
84 bytes from 172.16.1.10 icmp_seq=2 ttl=64 time=8.910 ms
84 bytes from 172.16.1.10 icmp_seq=3 ttl=64 time=10.462 ms
^C
VPCS> ping 172.16.2.10

172.16.2.10 icmp_seq=1 timeout
84 bytes from 172.16.2.10 icmp_seq=2 ttl=64 time=9.755 ms
84 bytes from 172.16.2.10 icmp_seq=3 ttl=64 time=5.210 ms
^C
VPCS> ping 172.16.4.10

84 bytes from 172.16.4.10 icmp_seq=1 ttl=64 time=16.622 ms
84 bytes from 172.16.4.10 icmp_seq=2 ttl=64 time=10.692 ms
^C
VPCS>

Nice. No problem there. Can we reach the ASAs?

VPCS> ping 172.16.1.1

84 bytes from 172.16.1.1 icmp_seq=1 ttl=255 time=120.722 ms
84 bytes from 172.16.1.1 icmp_seq=2 ttl=255 time=59.679 ms
^C
VPCS> ping 172.16.2.1

84 bytes from 172.16.2.1 icmp_seq=1 ttl=255 time=198.976 ms
84 bytes from 172.16.2.1 icmp_seq=2 ttl=255 time=35.903 ms
^C
VPCS> ping 172.16.4.1

172.16.4.1 icmp_seq=1 timeout
172.16.4.1 icmp_seq=2 timeout
172.16.4.1 icmp_seq=3 timeout
^C
VPCS>

So, we can reach the ASAs in Site1 and Site2 from PC3, but not Site4.

Site1 and Site2 are connected to Site3 through a VPN. Site3 and 4 are connected through a VPN and have the BGP link between them (the interfaces are 10.10.10.3/24 on Site3 and 10.10.10.4 on Site4).

Site3 knows about the 172.16.4.0/24 prefix through BGP:

Site3# sh route | b Gate
Gateway of last resort is 3.3.3.254 to network 0.0.0.0

S*      0.0.0.0 0.0.0.0 [1/0] via 3.3.3.254, Outside
C       3.3.3.0 255.255.255.0 is directly connected, Outside
L       3.3.3.3 255.255.255.255 is directly connected, Outside
C       10.10.10.0 255.255.255.0 is directly connected, BGP-Link
L       10.10.10.3 255.255.255.255 is directly connected, BGP-Link
C       172.16.3.0 255.255.255.0 is directly connected, Inside
L       172.16.3.1 255.255.255.255 is directly connected, Inside
B       172.16.4.0 255.255.255.0 [20/0] via 10.10.10.4, 1d20h

Site3# sh route 172.16.4.0 255.255.255.0

Routing entry for 172.16.4.0 255.255.255.0
  Known via "bgp 100", distance 20, metric 0
  Tag 200, type external
  Last update from 10.10.10.4 44:23:30 ago
  Routing Descriptor Blocks:
  * 10.10.10.4, from 10.10.10.4, 44:23:30 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 200
      MPLS label: no label string provided

Site3#

The “through” traffic works fine (PC3 to PC4), but not the “to” traffic (PC3 to Site4’s ASA).

We do not have un-equal routing, so this is not the issue:

PC4> ping 172.16.3.10

84 bytes from 172.16.3.10 icmp_seq=1 ttl=64 time=10.972 ms
84 bytes from 172.16.3.10 icmp_seq=2 ttl=64 time=8.927 ms
^C
PC4>

Site4# sh route 172.16.3.0 255.255.255.0

Routing entry for 172.16.3.0 255.255.255.0
  Known via "bgp 200", distance 20, metric 0
  Tag 100, type external
  Last update from 10.10.10.3 44:26:42 ago
  Routing Descriptor Blocks:
  * 10.10.10.3, from 10.10.10.3, 44:26:42 ago
     Route metric is 0, traffic share count is 1
     AS Hops 1
     Route tag 100
     MPLS label: no label string provided

Site4#

Likewise, PC4 cannot reach the Site3 ASA:

PC4> ping 172.16.3.1

172.16.3.1 icmp_seq=1 timeout
172.16.3.1 icmp_seq=2 timeout
172.16.3.1 icmp_seq=3 timeout
^C
PC4>

If we turn on “debug icmp trace”, we can see Site4 passing the traffic:

%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/11805 laddr 172.16.4.10/11805
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12317 laddr 172.16.4.10/12317
%ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/11805 laddr 172.16.4.10/11805
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12829 laddr 172.16.4.10/12829
%ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12317 laddr 172.16.4.10/12317
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13341 laddr 172.16.4.10/13341
%ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12829 laddr 172.16.4.10/12829
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13853 laddr 172.16.4.10/13853
%ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13341 laddr 172.16.4.10/13341
%ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13853 laddr 172.16.4.10/13853

Site3, however, shows the issue:

%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.4.10/541 to 172.16.3.1/0
%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.4.10/11805 to 172.16.3.1/0

If we try the ping from PC3 to Site4, we see the inverse:

%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42013 laddr 172.16.3.10/42013
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42525 laddr 172.16.3.10/42525
%ASA-6-302021: Teardown ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42013 laddr 172.16.3.10/42013
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/43037 laddr 172.16.3.10/43037
%ASA-6-302021: Teardown ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42525 laddr 172.16.3.10/42525

%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.3.10/42013 to 172.16.4.1/0

So, how do we fix this? The issue is that a route lookup is failing. With NAT, this is a pretty straightforward fix. If you have a large number of NAT statements, then it can be as easy as appending “route-lookup” to the NAT rule. Could this be the answer?

NAT and route-lookup

At the moment, we do not have any NAT rules, but the through traffic passes. Do we need to add a NAT rule, and perform route-lookup?

Let’s try it.

Site3(config)# nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup

Site4(config)# nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup

Nope:

%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.4.10/53792 to 172.16.3.1/0

%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.3.10/51744 to 172.16.4.1/0

Still no good.

The NAT is working though:

Site3(config)# sh nat
Manual NAT Policies (Section 1)
1 (Inside) to (BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup
 translate_hits = 5, untranslate_hits = 5
Site3(config)# 

Site4(config)# sh nat
Manual NAT Policies (Section 1)
1 (Inside) to (BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup
 translate_hits = 5, untranslate_hits = 5
Site4(config)#

Let’s check out packet tracer:

Site4# packet-tracer input BGP-Link icmp 172.16.3.10 7 0 172.16.4.1 det

Result:
input-interface: BGP-Link
input-status: up
input-line-status: up
%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.3.10/0 to 172.16.4.1/0
Action: drop
Drop-reason: (no-route) No route to host

Site4#

So, still can’t get from the BGP-Link to the management-side of the ASA in Site4.

Let’s change the NAT around a bit…

Site3(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup
%ASA-6-305010: Teardown static translation from Inside:172.16.3.0 to BGP-Link:172.16.3.0 duration 0:22:02
%ASA-6-305010: Teardown static translation from BGP-Link:172.16.4.0 to Inside:172.16.4.0 duration 0:22:02
Site3(config)# nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup
Site3(config)#

Site4(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup
%ASA-6-305010: Teardown static translation from Inside:172.16.4.0 to BGP-Link:172.16.4.0 duration 0:23:18
%ASA-6-305010: Teardown static translation from BGP-Link:172.16.3.0 to Inside:172.16.3.0 duration 0:23:18
Site4(config)# nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup
Site4(config)#

The result is the same. So let’s remove NAT completely.

Site3(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup
Site3(config)# 

Site4(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup
Site4(config)#

OK, so what about trying to use the management interface?

Remote management through the management interface.

Site4(config)# int management 0/0
Site4(config-if)# ip address 40.40.40.4 255.255.255.0 
Site4(config-if)# management-only 
Site4(config-if)# no shut
Site4(config-if)# router bgp 200
Site4(config-router)# address-family ipv4 unicast
Site4(config-router-af)# network 40.40.40.0 mask 255.255.255.0
Site4(config-router-af)#end
Site4# clear bgp *

BGP table version is 3, local router ID is 10.10.10.4
Network          Next Hop  Metric LocPrf Weight Path
*> 172.16.4.0/24  0.0.0.0       0         32768    i

Total number of prefixes 1 
Site4#

Doesn’t get advertised. This is because a management-only interface does not participate in routing. So, what about removing the management-only command?

Site4(config)# int management 0/0
Site4(config-if)# no management-only
Site4(config-if)# 
Site4(config-if)# icmp permit any ?

configure mode commands/options:
 <0-255> Enter ICMP type number (0 - 255)
 echo 
 echo-reply 
 time-exceeded 
 unreachable 
Current available interface(s):
 BGP-Link Name of interface GigabitEthernet0/5
 Inside Name of interface GigabitEthernet0/1
 Outside Name of interface GigabitEthernet0/0
Site4(config-if)# icmp permit any

Wouldn’t be able to ping it anyway.

So, I am out of ideas at this stage. If you can fix it, please let me know.

Next step is to open a TAC case.

Configs

You can download a zip file with the ASA configs below.

ASA-BGP-Management

Updates

  1. Swapping BGP for EIGRP gets the same result. Therefore it is not routing protocol specific.
  2. Performing a shut on Gi0/5 forces the traffic to flow over the VPN, and PC3 can reach the Site4 ASA.

7 Comments

  1. greensboro April 9, 2017
    • Stuart Fordham April 9, 2017
  2. rjscciesecuritylab April 10, 2017
  3. Walter Lopez April 11, 2017
  4. Rj Pitts April 11, 2017
    • Stuart Fordham April 11, 2017
  5. Darin Luckie June 9, 2017

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.