It’s not been a fun week. Last Monday, I started to make some changes to our firewalls that would allow me (this coming Monday) to move some equipment from one part of our network to a new, locked-down part of the network. Because PCI. Little did I know, I would enter the hell that is “no valid adjacency”.

As I was not actually moving the equipment that day, I could only use Cisco’s packet-tracer on the ASA to test whether or not the traffic would work as intended. It should be noted that this is in a remote site, and should just need to make some VLAN changes on the Catalysts, once the vendor logs in and changes the IP addresses on the interfaces. This would mean that the vendor equipment would be in its own part of the network, separated by firewall rules from the rest of the network.

Sounds easy enough.

Packet-tracer has been around since 7.2(1) and is a go-to staple of troubleshooting. It’s got me out of many a tricky situation. It makes sense that I would try and validate the changes before I move the equipment. From the Cisco website:

The packet-tracer command can be used in privileged EXEC mode to generate a 5-to-6 tuple packet against a firewall’s current configurations. For clarity, the packet-tracer syntax is shown separately for ICMP, TCP/UDP/SCTP, and IP packet modeling.

https://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/I-R/cmdref2/p1.html

The packet-tracer command provides detailed information about the packets and how they are processed by the ASA. packet-tracer allows a firewall administrator to inject a virtual packet into the security appliance and track the flow from ingress to egress.

https://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/I-R/cmdref2/p1.html

There are a couple of really important things to note here, which I have bolded, but to put it in a one-liner:

We can use packet-tracer to generate a virtual packet.

Remember this, it’s a virtual packet. This is important.

So the interfaces were created, ACLs created, NAT rules put in place and then tested with packet-tracer.

That’s where things went weird.

Let’s start with the relevant interfaces and rules:

interface GigabitEthernet0/2.13  
  description SW  
  vlan 13  
  nameif SW-VLAN-13  
  security-level 75  
  ip address 10.100.13.1 255.255.255.0 standby 10.100.13.3

We have a subinterface tagged with VLAN 13. The interface is up, and ping-able from an SVI on the catalyst:

SW02#ping 10.100.13.1 so vl 13
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.100.13.1, timeout is 2 seconds:
Packet sent with a source address of 10.100.13.5
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
SW02#

We also have an ACL permitting some traffic:

 access-list DMZ-VLAN-11_access_in extended permit object-group DM_INLINE_SERVICE_9 10.100.11.0 255.255.255.0 host 10.100.13.155  

For future reference, the 10.100.11.0/24 network is in the DMZ-NETWORKS object group. DM_INLINE_SERVICE_9 is for rsh and syslog:

object-group service DM_INLINE_SERVICE_9  
 service-object tcp destination eq rsh   
 service-object udp destination eq syslog  

So that we don’t do any NAT translations, we have a NAT exemption:

nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp

So, we should be able to send syslogs from any host in the 10.100.11.0/24 network to the 10.100.13.155 host. Simple, right?

Let’s test this using the packet-tracer, starting with the command:

packet-tracer input DMZ-VLAN-11 udp 10.100.11.31 syslog 10.100.13.155 syslog detailed

So here we are simulating a syslog packet being sent from 10.100.11.31 to 10.100.13.155. Let’s see how far the packet-tracer gets.

Phase: 1
 Type: UN-NAT
 Subtype: static
 Result: ALLOW
 Config:
 nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp description SW access to DMZ
 Additional Information:
 NAT divert to egress interface SW-VLAN-13
 Untranslate 10.100.13.155/514 to 10.100.13.155/514

We pass the UN-NAT. So our IP addresses will remain unchanged. We can also see which interface the packet will egress.

Phase: 2
 Type: ACCESS-LIST
 Subtype: log
 Result: ALLOW
 Config:
 access-group DMZ-VLAN-11_access_in in interface DMZ-VLAN-11
 access-list DMZ-VLAN-11_access_in extended permit object-group DM_INLINE_SERVICE_9 10.100.11.0 255.255.255.0 host 10.100.13.155
 object-group service DM_INLINE_SERVICE_9
  service-object tcp destination eq rsh
  service-object udp destination eq syslog
 Additional Information:
  Forward Flow based lookup yields rule:
  in  id=0x7ff076dff220, priority=13, domain=permit, deny=false
         hits=18, user_data=0x7ff06bb01c40, cs_id=0x0, use_real_addr, flags=0x0, protocol=17
         src ip/id=10.100.11.0, mask=255.255.255.0, port=0, tag=any
         dst ip/id=10.100.13.155, mask=255.255.255.255, port=514, tag=any, dscp=0x0
         input_ifc=DMZ-VLAN-11, output_ifc=any

We pass the access-list.

Phase: 3
 Type: NAT
 Subtype:
 Result: ALLOW
 Config:
 nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp
 Additional Information:
 Static translate 10.100.11.31/514 to 10.100.11.31/514
  Forward Flow based lookup yields rule:
  in  id=0x7ff07a7f3bb0, priority=6, domain=nat, deny=false
         hits=1, user_data=0x7ff0767ed220, cs_id=0x0, flags=0x0, protocol=0
         src ip/id=10.100.11.0, mask=255.255.255.0, port=0, tag=any
         dst ip/id=10.100.13.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
         input_ifc=DMZ-VLAN-11, output_ifc=SW-VLAN-13

We pass another NAT phase, but we can see that we now know where the packet is coming from, and which interface it will be sent out of.

Phase: 4
 Type: NAT
 Subtype: per-session
 Result: ALLOW
 Config:
 Additional Information:
  Forward Flow based lookup yields rule:
  in  id=0x7ff075863870, priority=0, domain=nat-per-session, deny=true
         hits=19434970, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
         src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
         dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
         input_ifc=any, output_ifc=any

We pass another NAT.

Phase: 5
 Type: IP-OPTIONS
 Subtype:
 Result: ALLOW
 Config:
 Additional Information:
  Forward Flow based lookup yields rule:
  in  id=0x7ff076a17d20, priority=0, domain=inspect-ip-options, deny=true
         hits=123529407, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
         src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
         dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
         input_ifc=DMZ-VLAN-11, output_ifc=any

We pass through an IP options phase.

Phase: 6
 Type: FOVER
 Subtype: standby-update
 Result: ALLOW
 Config:
 Additional Information:
  Forward Flow based lookup yields rule:
  in  id=0x7ff076a58f10, priority=20, domain=lu, deny=false
         hits=1337465, user_data=0x0, cs_id=0x0, flags=0x0, protocol=17
         src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
         dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
         input_ifc=DMZ-VLAN-11, output_ifc=any

We pass through the Failover phase.

Phase: 7
 Type: NAT
 Subtype: rpf-check
 Result: ALLOW
 Config:
 nat (SW-VLAN-13,DMZ-VLAN-11) source static obj-10.100.13.0 obj-10.100.13.0 destination static DMZ-NETWORKS DMZ-NETWORKS no-proxy-arp
 Additional Information:
  Forward Flow based lookup yields rule:
  out id=0x7ff07a7f3f90, priority=6, domain=nat-reverse, deny=false
         hits=2, user_data=0x7ff07d0e1840, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
         src ip/id=10.100.11.0, mask=255.255.255.0, port=0, tag=any
         dst ip/id=10.100.13.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
         input_ifc=DMZ-VLAN-11, output_ifc=SW-VLAN-13

We pass through the reverse-path forwarding phase.

Phase: 8
 Type: NAT
 Subtype: per-session
 Result: ALLOW
 Config:
 Additional Information:
  Reverse Flow based lookup yields rule:
  in  id=0x7ff075863870, priority=0, domain=nat-per-session, deny=true
         hits=19434972, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
         src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
         dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
         input_ifc=any, output_ifc=any

Another NAT phase is successfully passed.

Phase: 9
 Type: IP-OPTIONS
 Subtype:
 Result: ALLOW
 Config:
 Additional Information:
  Reverse Flow based lookup yields rule:
  in  id=0x7ff07628bdc0, priority=0, domain=inspect-ip-options, deny=true
         hits=35, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
         src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
         dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
         input_ifc=SW-VLAN-13, output_ifc=any

Some more IP-Options.

Phase: 10
 Type: FLOW-CREATION
 Subtype:
 Result: ALLOW
 Config:
 Additional Information:
 New flow created with id 124954958, packet dispatched to next module
 Module information for forward flow …
 snp_fp_inspect_ip_options
 snp_fp_translate
 snp_fp_adjacency
 snp_fp_fragment
 snp_fp_tracer_drop
 snp_ifc_stat

Module information for reverse flow …
 snp_fp_inspect_ip_options
 snp_fp_translate
 snp_fp_adjacency
 snp_fp_fragment
 snp_fp_tracer_drop
 snp_ifc_stat

The flow gets created.

Phase: 11
 Type: ROUTE-LOOKUP
 Subtype: Resolve Egress Interface
 Result: ALLOW
 Config:
 Additional Information:
 found next-hop 10.100.13.155 using egress ifc  SW-VLAN-13

The route is checked and we know which interface to pass the packet out of.

So each check has allowed the packet. We aren’t getting NATd, and we are not being denied any ACLs, so it looks golden, right? Let’s check the result.

Result:
 input-interface: DMZ-VLAN-11
 input-status: up
 input-line-status: up
 output-interface: SW-VLAN-13
 output-status: up
 output-line-status: up
 Action: drop
 Drop-reason: (no-adjacency) No valid adjacency

Balls. 

No valid adjacency usually means that we either don’t know where to send the packet, or the destination won’t be able to tell where the packet came from.

Let’s have a brief recap. We created a virtual packet, passed it through the firewall, where it passed each and every check, but still, the packet gets dropped. We have already proved (by pinging the firewall interface from the switch SVI) that we have physical connectivity. The packet tracer has picked up the correct ACL, ingress and egress interfaces, we are not NATting either the source or destination address, and all the phases have passed, only for us to fall at the final hurdle. Bizzare, yes?

So let’s go through some of the troubleshooting steps I did.

• I tested Inside to SW-VLAN-13. Same error.
• In case there was an issue with the 10.100.0.0/16 route conflicting, I changed the 10.100.13.0/.24 subnet to 10.101.13.0/24. Same error.
• I changed the NAT around, to match how DMZ-VLAN-11 to Inside works. Same error.
• Moved the NAT statements around. Same error.
• Failed over the firewalls. Same error.

Eventually, I got in contact with Cisco TAC. I thought I was hitting a known bug (which is a critical version 9.x flaw, resulting in “no valid adjacency” when there is a routing change), and they could either confirm or deny this.

The nice guy from Cisco did some checks, moved some things around, did some more checks, then added an ACL.

The ACL he added permitted syslog from the 10.100.11.0/24 network to the 10.100.13.5 SVI on the catalyst. It’s the only thing in that part of the network which we had to test with.

It worked. This “virtual” packet worked.

How frustrating.

Therefore, even though packet-tracer simulates a packet, a virtual packet, it still needs an ARP entry for the trace to work. Otherwise, we get the “No valid adjacency” error (at least in the 9.10(1) version I am running).

That was a fun week.