Picture the scene…
We have four sites. Each site has a VPN to each of the other sites, forming a mesh, with default routes pointing to the vIOS router.
Traffic from each of the PCs to the other PCs works fine. All is good, right?
Here are the pings from PC3:
VPCS> ping 172.16.1.10 172.16.1.10 icmp_seq=1 timeout 84 bytes from 172.16.1.10 icmp_seq=2 ttl=64 time=8.910 ms 84 bytes from 172.16.1.10 icmp_seq=3 ttl=64 time=10.462 ms ^C VPCS> ping 172.16.2.10 172.16.2.10 icmp_seq=1 timeout 84 bytes from 172.16.2.10 icmp_seq=2 ttl=64 time=9.755 ms 84 bytes from 172.16.2.10 icmp_seq=3 ttl=64 time=5.210 ms ^C VPCS> ping 172.16.4.10 84 bytes from 172.16.4.10 icmp_seq=1 ttl=64 time=16.622 ms 84 bytes from 172.16.4.10 icmp_seq=2 ttl=64 time=10.692 ms ^C VPCS>
Nice. No problem there. Can we reach the ASAs?
VPCS> ping 172.16.1.1 84 bytes from 172.16.1.1 icmp_seq=1 ttl=255 time=120.722 ms 84 bytes from 172.16.1.1 icmp_seq=2 ttl=255 time=59.679 ms ^C VPCS> ping 172.16.2.1 84 bytes from 172.16.2.1 icmp_seq=1 ttl=255 time=198.976 ms 84 bytes from 172.16.2.1 icmp_seq=2 ttl=255 time=35.903 ms ^C VPCS> ping 172.16.4.1 172.16.4.1 icmp_seq=1 timeout 172.16.4.1 icmp_seq=2 timeout 172.16.4.1 icmp_seq=3 timeout ^C VPCS>
So, we can reach the ASAs in Site1 and Site2 from PC3, but not Site4.
Site1 and Site2 are connected to Site3 through a VPN. Site3 and 4 are connected through a VPN and have the BGP link between them (the interfaces are 10.10.10.3/24 on Site3 and 10.10.10.4 on Site4).
Site3 knows about the 172.16.4.0/24 prefix through BGP:
Site3# sh route | b Gate
Gateway of last resort is 3.3.3.254 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 3.3.3.254, Outside
C 3.3.3.0 255.255.255.0 is directly connected, Outside
L 3.3.3.3 255.255.255.255 is directly connected, Outside
C 10.10.10.0 255.255.255.0 is directly connected, BGP-Link
L 10.10.10.3 255.255.255.255 is directly connected, BGP-Link
C 172.16.3.0 255.255.255.0 is directly connected, Inside
L 172.16.3.1 255.255.255.255 is directly connected, Inside
B 172.16.4.0 255.255.255.0 [20/0] via 10.10.10.4, 1d20h
Site3# sh route 172.16.4.0 255.255.255.0
Routing entry for 172.16.4.0 255.255.255.0
Known via "bgp 100", distance 20, metric 0
Tag 200, type external
Last update from 10.10.10.4 44:23:30 ago
Routing Descriptor Blocks:
* 10.10.10.4, from 10.10.10.4, 44:23:30 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 200
MPLS label: no label string provided
Site3#
The “through” traffic works fine (PC3 to PC4), but not the “to” traffic (PC3 to Site4’s ASA).
We do not have un-equal routing, so this is not the issue:
PC4> ping 172.16.3.10
84 bytes from 172.16.3.10 icmp_seq=1 ttl=64 time=10.972 ms
84 bytes from 172.16.3.10 icmp_seq=2 ttl=64 time=8.927 ms
^C
PC4>
Site4# sh route 172.16.3.0 255.255.255.0
Routing entry for 172.16.3.0 255.255.255.0
Known via "bgp 200", distance 20, metric 0
Tag 100, type external
Last update from 10.10.10.3 44:26:42 ago
Routing Descriptor Blocks:
* 10.10.10.3, from 10.10.10.3, 44:26:42 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 100
MPLS label: no label string provided
Site4#
Likewise, PC4 cannot reach the Site3 ASA:
PC4> ping 172.16.3.1 172.16.3.1 icmp_seq=1 timeout 172.16.3.1 icmp_seq=2 timeout 172.16.3.1 icmp_seq=3 timeout ^C PC4>
If we turn on “debug icmp trace”, we can see Site4 passing the traffic:
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/11805 laddr 172.16.4.10/11805 %ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12317 laddr 172.16.4.10/12317 %ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/11805 laddr 172.16.4.10/11805 %ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12829 laddr 172.16.4.10/12829 %ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12317 laddr 172.16.4.10/12317 %ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13341 laddr 172.16.4.10/13341 %ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/12829 laddr 172.16.4.10/12829 %ASA-6-302020: Built outbound ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13853 laddr 172.16.4.10/13853 %ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13341 laddr 172.16.4.10/13341 %ASA-6-302021: Teardown ICMP connection for faddr 172.16.3.1/0 gaddr 172.16.4.10/13853 laddr 172.16.4.10/13853
Site3, however, shows the issue:
%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.4.10/541 to 172.16.3.1/0 %ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.4.10/11805 to 172.16.3.1/0
If we try the ping from PC3 to Site4, we see the inverse:
%ASA-6-302020: Built outbound ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42013 laddr 172.16.3.10/42013 %ASA-6-302020: Built outbound ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42525 laddr 172.16.3.10/42525 %ASA-6-302021: Teardown ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42013 laddr 172.16.3.10/42013 %ASA-6-302020: Built outbound ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/43037 laddr 172.16.3.10/43037 %ASA-6-302021: Teardown ICMP connection for faddr 172.16.4.1/0 gaddr 172.16.3.10/42525 laddr 172.16.3.10/42525 %ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.3.10/42013 to 172.16.4.1/0
So, how do we fix this? The issue is that a route lookup is failing. With NAT, this is a pretty straightforward fix. If you have a large number of NAT statements, then it can be as easy as appending “route-lookup” to the NAT rule. Could this be the answer?
NAT and route-lookup
At the moment, we do not have any NAT rules, but the through traffic passes. Do we need to add a NAT rule, and perform route-lookup?
Let’s try it.
Site3(config)# nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup Site4(config)# nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup
Nope:
%ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.4.10/53792 to 172.16.3.1/0 %ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.3.10/51744 to 172.16.4.1/0
Still no good.
The NAT is working though:
Site3(config)# sh nat Manual NAT Policies (Section 1) 1 (Inside) to (BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup translate_hits = 5, untranslate_hits = 5 Site3(config)# Site4(config)# sh nat Manual NAT Policies (Section 1) 1 (Inside) to (BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup translate_hits = 5, untranslate_hits = 5 Site4(config)#
Let’s check out packet tracer:
Site4# packet-tracer input BGP-Link icmp 172.16.3.10 7 0 172.16.4.1 det Result: input-interface: BGP-Link input-status: up input-line-status: up %ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.3.10/0 to 172.16.4.1/0 Action: drop Drop-reason: (no-route) No route to host Site4#
So, still can’t get from the BGP-Link to the management-side of the ASA in Site4.
Let’s change the NAT around a bit…
Site3(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup %ASA-6-305010: Teardown static translation from Inside:172.16.3.0 to BGP-Link:172.16.3.0 duration 0:22:02 %ASA-6-305010: Teardown static translation from BGP-Link:172.16.4.0 to Inside:172.16.4.0 duration 0:22:02 Site3(config)# nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup Site3(config)# Site4(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup %ASA-6-305010: Teardown static translation from Inside:172.16.4.0 to BGP-Link:172.16.4.0 duration 0:23:18 %ASA-6-305010: Teardown static translation from BGP-Link:172.16.3.0 to Inside:172.16.3.0 duration 0:23:18 Site4(config)# nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup Site4(config)#
The result is the same. So let’s remove NAT completely.
Site3(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.3.0 OBJ-172.16.3.0 destination static OBJ-172.16.3.0 OBJ-172.16.3.0 no-proxy-arp route-lookup Site3(config)# Site4(config)# no nat (Inside,BGP-Link) source static OBJ-172.16.4.0 OBJ-172.16.4.0 destination static OBJ-172.16.4.0 OBJ-172.16.4.0 no-proxy-arp route-lookup Site4(config)#
OK, so what about trying to use the management interface?
Remote management through the management interface.
Site4(config)# int management 0/0 Site4(config-if)# ip address 40.40.40.4 255.255.255.0 Site4(config-if)# management-only Site4(config-if)# no shut Site4(config-if)# router bgp 200 Site4(config-router)# address-family ipv4 unicast Site4(config-router-af)# network 40.40.40.0 mask 255.255.255.0 Site4(config-router-af)#end Site4# clear bgp * BGP table version is 3, local router ID is 10.10.10.4 Network Next Hop Metric LocPrf Weight Path *> 172.16.4.0/24 0.0.0.0 0 32768 i Total number of prefixes 1 Site4#
Doesn’t get advertised. This is because a management-only interface does not participate in routing. So, what about removing the management-only command?
Site4(config)# int management 0/0 Site4(config-if)# no management-only Site4(config-if)# Site4(config-if)# icmp permit any ? configure mode commands/options: <0-255> Enter ICMP type number (0 - 255) echo echo-reply time-exceeded unreachable Current available interface(s): BGP-Link Name of interface GigabitEthernet0/5 Inside Name of interface GigabitEthernet0/1 Outside Name of interface GigabitEthernet0/0 Site4(config-if)# icmp permit any
Wouldn’t be able to ping it anyway.
So, I am out of ideas at this stage. If you can fix it, please let me know.
Next step is to open a TAC case.
Configs
You can download a zip file with the ASA configs below.
Updates
- Swapping BGP for EIGRP gets the same result. Therefore it is not routing protocol specific.
- Performing a shut on Gi0/5 forces the traffic to flow over the VPN, and PC3 can reach the Site4 ASA.
Hmm, So with the link up(I’m thinking it’s GI0/6 going by the diagram) traffic from PC3 to ASA4 is going over the tunnel and then trying to take BGP back to site 3 it would seem or asymmetric. I wonder if we can boil it down to just BGP and remove the VPN piece all together.
I tried your idea. Remove the VPN config for the other site from both ASAs. Still get the same error, but this does completely eliminate the possibility of asymmetric routing… If I shut both the Gi0/0 interfaces (therefore they can only take the BGP link), still get the same issue:
Site4(config-if)# int gi 0/0
Site4(config-if)# shut
%ASA-4-411004: Interface Outside, changed state to administratively down
%ASA-4-411004: Interface GigabitEthernet0/0, changed state to administratively down
Site4(config-if)# sh route
Gateway of last resort is not set
C 10.10.10.0 255.255.255.0 is directly connected, BGP-Link
L 10.10.10.4 255.255.255.255 is directly connected, BGP-Link
B 172.16.3.0 255.255.255.0 [20/0] via 10.10.10.3, 00:04:00
C 172.16.4.0 255.255.255.0 is directly connected, Inside
L 172.16.4.1 255.255.255.255 is directly connected, Inside
Site4(config-if)# %ASA-6-110002: Failed to locate egress interface for ICMP from BGP-Link:172.16.3.10/53886 to 172.16.4.1/0
Again, through traffic (PC3 to PC4) works fine…
Stuart, I think this is excepted. I did some research on the “management-access” command and it appears this command only allows access to the inside interface for ping and other services when you use VPNs. I tested with my 5512x’s for both L2L and Remote access and saw the same behavior. This would explain why it works for the other sites but fails between site 3 and site 4. You confirmed this again with the EIGRP test.
In short, it appears that NON-VPN “to the box traffic” is not allowed on inside or higher security level interfaces. Let me know what you think and or what TAC says.
I agree with ‘rjscciesecuritylab’, the “management-access” command will only work if the management traffic comes through a VPN.
It is expected that you can’t (out of a VPN) access an interface that is on the other side of the firewall, for example:
1. If you are on the inside interface, you CAN’T access the outside.
2. If you are on the outside interface, you CAN’T access the inside.
Both points above are by default and is how the developers decided to manage the traffic to the ASA. At the end why in the world if you want to access the ASA you want to go to the far interface instead of the nearest?
I think the idea was to manage the ASA’s remotely. In this case, having a jump box you can remote into then hit the inside interface maybe the only solution if you connecting over a NON-VPN connection.
I got a response back from Cisco:
“You are right that this behavior seems to be expected. ASA works slightly different than routers and other Cisco devices. It is required to enhance the security of firewall. On ASA you can’t access the inside interface of the ASA from the outside. In general you can’t access other interfaces than the one through which you connect to ASA. The only exception is the situation when you connect to the device over the VPN tunnel and then you skip the inbound ACL verification. So in general any attempt to connect, or event ping the inside interface form outside will fail.”
Makes sense. Bit frustrating when trying to set up Solarwinds, but I worked around it by setting that datacenter to use the external address (same one that would be used by the VPN) instead of the internal address.
Forgive my ignorance about ASA’s, but based on your output, it would seem to me that when you removed NAT from Sites 3&4, you didn’t do the same for sites 1&2…but also, I had an eerily similar situation between my FreeBSD PF Firewall and all over traffic to an ESXi host. Stupid question maybe, but why would you have NAT on private-private addresses? This causes double NAT on egress of the private space, which is never good 😛
Maybe I am mistaken, but it seems to be a filtering problem on the ASA…as Cisco said, due to secuity. For me, I have the following:
Home (Cable) > OpenVPN (regular Windows FW)
AWS Cloud OpenVPN subnet (172.30.0.0/16)
Colo site (VMWare ESXi private cloud) (10.10.15.0/24)
For a whole day I struggled with why ICMP worked but TCP/UDP did not. routing is fine:
[[email protected] ~]# cat /etc/openvpn/ccd/cloud
ifconfig-push 172.30.0.26 172.30.0.25
iroute 10.10.15.0 255.255.255.0
I was wondering about the ESXi firewall but knew this was not the issue since (as in your case) traffic passes THROUGH IT. Finally, on a whim, I added:
[[email protected] ~]# egrep “^pass quick” /etc/pf.conf
pass quick on $int_if inet all keep state
pass quick on $vpn_if inet all keep state
To the colo VM FW (FreeBSD PF) and now:
[[email protected] ~]$ sudo nmap -T4 10.10.15.254
Starting Nmap 7.40 ( https://nmap.org ) at 2017-06-09 00:04 EDT
Warning: 10.10.15.254 giving up on port because retransmission cap hit (6).
Nmap scan report for 10.10.15.254
Host is up (0.036s latency).
Not shown: 993 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
902/tcp open iss-realsecure
8000/tcp open http-alt
8300/tcp open tmi
9080/tcp open glrpc
Cheers,
P.S I bought your BGP book as I am studying for my CCIE SP and would be VERY interested if you do share your solution when you find it!