It’s not been a fun week. Last Monday, I started to make some changes to our firewalls that would allow me (this coming Monday) to move some equipment from one part of our network to a new, locked-down part of the network. Because PCI. Little did I know, I would enter the hell that is “no valid adjacency”.
As I was not actually moving the equipment that day, I could only use Cisco’s packet-tracer on the ASA to test whether or not the traffic would work as intended. It should be noted that this is in a remote site, and should just need to make some VLAN changes on the Catalysts, once the vendor logs in and changes the IP addresses on the interfaces. This would mean that the vendor equipment would be in its own part of the network, separated by firewall rules from the rest of the network.
Sounds easy enough.
Packet-tracer has been around since 7.2(1) and is a go-to staple of troubleshooting. It’s got me out of many a tricky situation. It makes sense that I would try and validate the changes before I move the equipment. From the Cisco website:
The packet-tracer command can be used in privileged EXEC mode to generate a 5-to-6 tuple packet against a firewall’s current configurations. For clarity, the packet-tracer syntax is shown separately for ICMP, TCP/UDP/SCTP, and IP packet modeling.
https://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/I-R/cmdref2/p1.html
The packet-tracer command provides detailed information about the packets and how they are processed by the ASA. packet-tracer allows a firewall administrator to inject a virtual packet into the security appliance and track the flow from ingress to egress.
https://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/I-R/cmdref2/p1.html
There are a couple of really important things to note here, which I have bolded, but to put it in a one-liner:
We can use packet-tracer to generate a virtual packet.
Remember this, it’s a virtual packet. This is important.
So the interfaces were created, ACLs created, NAT rules put in place and then tested with packet-tracer.
That’s where things went weird.
Let’s start with the relevant interfaces and rules:
interface GigabitEthernet0/2.13
description SW
vlan 13
nameif SW-VLAN-13
security-level 75
ip address 10.100.13.1 255.255.255.0 standby 10.100.13.3
We have a subinterface tagged with VLAN 13. The interface is up, and ping-able from an SVI on the catalyst:
SW02#ping 10.100.13.1 so vl 13
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.100.13.1, timeout is 2 seconds:
Packet sent with a source address of 10.100.13.5
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
SW02#
We also have an ACL permitting some traffic:
access-list DMZ-VLAN-11_access_in extended permit object-group DM_INLINE_SERVICE_9 10.100.11.0 255.255.255.0 host 10.100.13.155
For future reference, the 10.100.11.0/24 network is in the DMZ-NETWORKS object group. DM_INLINE_SERVICE_9 is for rsh and syslog:
object-group service DM_INLINE_SERVICE_9
service-object tcp destination eq rsh
service-object udp destination eq syslog
So that we don’t do any NAT translations, we have a NAT exemption:
nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp
So, we should be able to send syslogs from any host in the 10.100.11.0/24 network to the 10.100.13.155 host. Simple, right?
Let’s test this using the packet-tracer, starting with the command:
packet-tracer input DMZ-VLAN-11 udp 10.100.11.31 syslog 10.100.13.155 syslog detailed
So here we are simulating a syslog packet being sent from 10.100.11.31 to 10.100.13.155. Let’s see how far the packet-tracer gets.
Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp description SW access to DMZ
Additional Information:
NAT divert to egress interface SW-VLAN-13
Untranslate 10.100.13.155/514 to 10.100.13.155/514
We pass the UN-NAT. So our IP addresses will remain unchanged. We can also see which interface the packet will egress.
Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group DMZ-VLAN-11_access_in in interface DMZ-VLAN-11
access-list DMZ-VLAN-11_access_in extended permit object-group DM_INLINE_SERVICE_9 10.100.11.0 255.255.255.0 host 10.100.13.155
object-group service DM_INLINE_SERVICE_9
service-object tcp destination eq rsh
service-object udp destination eq syslog
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff076dff220, priority=13, domain=permit, deny=false
hits=18, user_data=0x7ff06bb01c40, cs_id=0x0, use_real_addr, flags=0x0, protocol=17
src ip/id=10.100.11.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=10.100.13.155, mask=255.255.255.255, port=514, tag=any, dscp=0x0
input_ifc=DMZ-VLAN-11, output_ifc=any
We pass the access-list.
Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp
Additional Information:
Static translate 10.100.11.31/514 to 10.100.11.31/514
Forward Flow based lookup yields rule:
in id=0x7ff07a7f3bb0, priority=6, domain=nat, deny=false
hits=1, user_data=0x7ff0767ed220, cs_id=0x0, flags=0x0, protocol=0
src ip/id=10.100.11.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=10.100.13.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
input_ifc=DMZ-VLAN-11, output_ifc=SW-VLAN-13
We pass another NAT phase, but we can see that we now know where the packet is coming from, and which interface it will be sent out of.
Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff075863870, priority=0, domain=nat-per-session, deny=true
hits=19434970, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any
We pass another NAT.
Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff076a17d20, priority=0, domain=inspect-ip-options, deny=true
hits=123529407, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=DMZ-VLAN-11, output_ifc=any
We pass through an IP options phase.
Phase: 6
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7ff076a58f10, priority=20, domain=lu, deny=false
hits=1337465, user_data=0x0, cs_id=0x0, flags=0x0, protocol=17
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=DMZ-VLAN-11, output_ifc=any
We pass through the Failover phase.
Phase: 7
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7ff07a7f3f90, priority=6, domain=nat-reverse, deny=false
hits=2, user_data=0x7ff07d0e1840, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.100.11.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=10.100.13.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
input_ifc=DMZ-VLAN-11, output_ifc=SW-VLAN-13
We pass through the reverse-path forwarding phase.
Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7ff075863870, priority=0, domain=nat-per-session, deny=true
hits=19434972, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any
Another NAT phase is successfully passed.
Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7ff07628bdc0, priority=0, domain=inspect-ip-options, deny=true
hits=35, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=SW-VLAN-13, output_ifc=any
Some more IP-Options.
Phase: 10
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 124954958, packet dispatched to next module
Module information for forward flow …
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat
Module information for reverse flow …
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat
The flow gets created.
Phase: 11
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 10.100.13.155 using egress ifc SW-VLAN-13
The route is checked and we know which interface to pass the packet out of.
So each check has allowed the packet. We aren’t getting NATd, and we are not being denied any ACLs, so it looks golden, right? Let’s check the result.
Result:
input-interface: DMZ-VLAN-11
input-status: up
input-line-status: up
output-interface: SW-VLAN-13
output-status: up
output-line-status: up
Action: drop
Drop-reason: (no-adjacency) No valid adjacency
Balls.
No valid adjacency usually means that we either don’t know where to send the packet, or the destination won’t be able to tell where the packet came from.
Let’s have a brief recap. We created a virtual packet, passed it through the firewall, where it passed each and every check, but still, the packet gets dropped. We have already proved (by pinging the firewall interface from the switch SVI) that we have physical connectivity. The packet tracer has picked up the correct ACL, ingress and egress interfaces, we are not NATting either the source or destination address, and all the phases have passed, only for us to fall at the final hurdle. Bizzare, yes?
So let’s go through some of the troubleshooting steps I did.
- I tested Inside to SW-VLAN-13. Same error.
- In case there was an issue with the 10.100.0.0/16 route conflicting, I changed the 10.100.13.0/.24 subnet to 10.101.13.0/24. Same error.
- I changed the NAT around, to match how DMZ-VLAN-11 to Inside works. Same error.
- Moved the NAT statements around. Same error.
- Failed over the firewalls. Same error.
Eventually, I got in contact with Cisco TAC. I thought I was hitting a known bug (which is a critical version 9.x flaw, resulting in “no valid adjacency” when there is a routing change), and they could either confirm or deny this.
The nice guy from Cisco did some checks, moved some things around, did some more checks, then added an ACL.
The ACL he added permitted syslog from the 10.100.11.0/24 network to the 10.100.13.5 SVI on the catalyst. It’s the only thing in that part of the network which we had to test with.
It worked. This “virtual” packet worked.
How frustrating.
Therefore, even though packet-tracer simulates a packet, a virtual packet, it still needs an ARP entry for the trace to work. Otherwise, we get the “No valid adjacency” error (at least in the 9.10(1) version I am running).
That was a fun week.
hi mate.
so the TAC engineer added a ACL on catalys switch?
No, on the ASA
in simple language 🙂
allows access tcp/upd to subnet 10.100.11.0/24 from host 10.100.13.155
!
in that case your nat command should be in this order
!
nat (DMZ-VLAN-11,SW-VLAN-13) source static DMZ-NETWORKS DMZ-NETWORKS destination static obj-10.100.13.0 obj-10.100.13.0 no-proxy-arp
!
where i guess your both VLAN13 and VLAN11 have same security-level.
In the bullet points, I said that I switched the NAT statement around (to match what you suggest).
The point is, without a valid host at the other end to reply to populate the ARP cache, packet-tracer failed (even though it’s a simulator!).
thank you.
my pleasure mate!
Thanks just has a very similar problem on a Firepower 9300. The packet tracer kept saying drop. The arp table didn’t show the destination.
then when the client fixed their connected interface and the arp was created the packet tracer began to work.
nice.
En mi caso, este problema se está dando en un cluster de firewalls ASA 5508X, luego de migrar a la última versión 9.12(4)30, por la noche de ese mismo día he visto que una conexión dejó de funcionar. Tengo la regla NAT siguiente:
Interface origen: external
Origen: any
Destino: 192.168.10.10
Interface destino: production
NAT origen: any
NAT destino: 192.168.20.10
Bidireccional
La regla que ahora usa la NAT 192.168.20.10 y no debería es:
Interface origen: production; Origen; 192.168.20.10; Interface destino: Desarrollo; Destino: 172.20.10.10; puerto 3389;
Las direcciones IP 192.168.20.10 y 172.20.10.10 pertenecen a interfaces de mi firewall conectados directamente, no debería haber NAT, tampoco problemas de ruta, sin embargo, en los logs adicional a la regla del párrafo anterior, se observa una NAT, quedando así:
Interface origen: production; Origen; 192.168.20.10; Interface destino: Desarrollo; Destino: 172.20.10.10; puerto 3389; NAT Origen 192.168.10.10