Most networks have redundancy (or at least should do). This can be in the form of additional cables cross-connecting devices, dual power supplies going to different feeds and HSRP on switches. All of this is good stuff, but some things can easily be taken for granted.
Consider the following topology:
We have two ASAs in a failover pair (Outside interface 10.30.1.1 with standby ip 10.30.1.2), both connecting to a switch (10.30.1.254), that has a loopback interface assigned to it to simulate the internet (10.250.1.1). The ASAs also connect to a pair of switches, set up in an HSRP group (ASA inside interface 10.40.1.1, standby ip 10.40.1.10, switch vlan 40 ip 10.40.1.254). The topology looks like this:
It looks pretty redundant right? But let’s work through the config and see what could go wrong. We’ll start by setting up the ASAs in a failover pair.
Cisco ASA failover setup
On the first ASA we set up failover as follows:
ciscoasa(config)# enable password cisco ciscoasa(config)# hostname ASA1 ASA1(config)# failover ASA1(config)# failover lan unit primary ASA1(config)# failover lan interface FAILOVER GigabitEthernet0/3 INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces ASA1(config)# failover key fail0ver ASA1(config)# failover replication http ASA1(config)# failover link FAILOVER GigabitEthernet0/3 ASA1(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.1 ASA1(config)# int gi0/3 ASA1(config-if)# no shut ASA1(config-if)# exit ASA1(config)# exit ASA1# copy run start
The second ASA is similar:
ciscoasa(config)# enable password cisco ciscoasa(config)# hostname ASA2 ASA2(config)# failover ASA2(config)# failover lan unit secondary ASA2(config)# failover lan interface FAILOVER GigabitEthernet0/3 INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces ASA2(config)# failover key fail0ver ASA2(config)# failover replication http ASA2(config)# failover link FAILOVER GigabitEthernet0/3 ASA2(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.2 ASA2(config)# int gi0/3 ASA2(config-if)# no shut ASA2(config-if)#exit ASA2(config)#exit ASA2# copy run start
We can then save our configuration, so that the partner (or mate) firewall gets sent the active configuration:
ASA1# copy run start ASA1# Beginning configuration replication: Sending to mate. End Configuration Replication to mate
I do like to set the prompt to show me easily which firewall I am currently on:
ASA1# ASA1# conf t ASA1(config)# prompt hostname state ASA1/act(config)# exit ASA1/act# copy run start Source filename [running-config]? Cryptochecksum: 9f0c4260 95438d98 2d1439ef 0704bf77 2693 bytes copied in 3.310 secs (897 bytes/sec) ASA1/act#
Our partner ASA will now change to show its status in the prompt:
ASA1# ASA1/stby# Cryptochecksum: 0df25109 9f4fecb5 cebb9db5 93664d73 ASA1/stby# 2691 bytes copied in 3.230 secs (897 bytes/sec) ASA1/stby#
Now we can configure our connection to the “internet” and to the inside network:
ASA Outside (internet) connection
ASA1/act# conf t ASA1/act(config)# int Gi0/0 ASA1/act(config-if)# nameif OUTSIDE INFO: Security level for "OUTSIDE" set to 0 by default. ASA1/act(config-if)# ip address 10.30.1.1 255.255.255.0 standby 10.30.1.2 ASA1/act(config-if)# no shut ASA1/act(config-if)#exit ASA1/act(config)# exit ASA1/act# copy run start
ASA Inside connection
ASA1/act(config-if)# int gi 0/1 ASA1/act(config-if)# nameif INSIDE INFO: Security level for "INSIDE" set to 100 by default. ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10 ASA1/act(config-if)# no shut ASA1/act(config-if)# exit ASA1/act(config)# exit ASA1/act# copy run start
Now we have connections on the inside and outside:
Switch HSRP setup
I have covered HSRP a few times now, so won’t go into any details, but here is the configuration walkthrough:
Switch(config)#hostname SW1 SW1(config)#vlan 40 SW1(config-vlan)#exit SW1(config)#int vlan 40 SW1(config-if)#ip address 10.40.1.2 255.255.255.0 00:06:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down SW1(config-if)#standby 40 ip 10.40.1.254 SW1(config-if)#standby 40 pri 150 SW1(config-if)# 00:09:16: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down SW1(config-if)#exit 00:09:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to up SW1(config)#int fastEthernet 1/0/2 SW1(config-if)#switchport trunk encap dot1q SW1(config-if)#switchport mode trunk 00:09:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up SW1(config-if)# 00:10:10: %HSRP-6-STATECHANGE: Vlan40 Grp 40 state Standby -> Active SW1(config-if)# Switch(config)#hostname SW2 SW2(config)#vlan 40 SW2(config-vlan)#exit SW2(config)#int vlan 40 *Mar 1 00:06:56.997: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down SW2(config-if)#ip address 10.40.1.3 255.255.255.0 SW2(config-if)#standby 40 ip 10.40.1.254 SW2(config-if)#standby pri 90 SW2(config-if)#exit SW2(config)#int fa0/2 SW2(config-if)#switchport trunk encap dot1q SW2(config-if)#switchport mode trunk SW2(config-if)# *Mar 1 00:09:22.447: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down *Mar 1 00:09:25.517: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up *Mar 1 00:09:55.515: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up *Mar 1 00:10:36.561: %HSRP-5-STATECHANGE: Vlan40 Grp 40 state Speak -> Standby SW2(config-if)#
Switch to ASA configuration
SW1(config)#int fa1/0/1 SW1(config-if)#switchport mode access SW1(config-if)#switchport access vlan 40 SW1(config-if)#exit SW1(config)#exit SW1#ping 10.40.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds: .!!!! Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms SW2(config)#int fa0/1 SW2(config-if)#switchport mode access SW2(config-if)#switchport access vlan 40 SW2(config-if)#exit SW2(config)#exit SW2# *Mar 1 01:09:09.651: %SYS-5-CONFIG_I: Configured from console by console SW2#ping 10.40.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
ASAs to “Internet” Configuration
On the active ASA we need to set a route out to the internet:
ASA1/act# sh run | i route route OUTSIDE 0.0.0.0 0.0.0.0 10.30.1.254 1
Internet configuration
The internet is very simple, we have a loopback that will be our end-point for testing, and a vlan for the ASAs to communicate with:
3550-SW3#sh run interface Loopback0 ip address 10.250.1.1 255.255.255.0 ! interface FastEthernet0/1 switchport access vlan 10 switchport mode access ! interface FastEthernet0/11 switchport access vlan 10 switchport mode access ! interface Vlan10 ip address 10.30.1.254 255.255.255.0
NAT statements to allow access
To allow the inside to talk to the internet we need to set up NATing on the ASAs:
global (OUTSIDE) 1 interface nat (INSIDE) 1 10.40.1.0 255.255.255.0
Testing
Let’s see how out connectivity looks:
ASA1/act# ping 10.250.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms ASA1/stby> ping 10.250.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms SW1#ping 10.250.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms SW2#ping 10.250.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms SW2#
So far so good!
Forcing failover and simulating outages
We can simulate issues in a number of ways. I started by removing the power to SW1. This made SW2 go active for vlan 40. SW2 has a good connection to ASA2 so pings were 100% successful.
With ASA1 as the standby we can force a failover from ASA2:
ASA1/stby# failover active
Switching to Active
SW2#ping 10.250.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
....!
Success rate is 20 percent (1/5), round-trip min/avg/max = 1/1/1 ms
ASA1/act# Waiting for the earlier webvpn instance to terminate...
Previous instance shut down. Starting a new one.
Switching to Standby
At this point the firewalls failed back, due to the traffic generated by SW2, so that ASA2 became the active mate again. If ASA2 had the power removed then they would not be able to fail back, and therefore SW2 would have no way to route the traffic over to SW1 (which would be the active firewall), and therefore the inside of the network would lose all internet communication.
So we can survive one ASA power outage and one switch outage (ASA1 and SW1 or ASA2 and SW2) but not if we lose SW2 and ASA1 or SW1 and ASA2.
So how can we get better resiliency? We can do this with a cable from SW1 to ASA2 and from SW2 to ASA1. This does require some reconfiguration on the ASAs, but how can we do this? We certainly can’t have another physical interface on the ASAs with the same IP range, as we would get an error informing us of an IP overlap. They key work here is physical interface. We can however set up a different type of interface to achieve this.
Cisco ASA Redundant interfaces
Redundant interfaces are two or more interfaces joined together in a redundancy group. They act like physical interfaces, in that they have an IP address, but with the added bonus of having multiple links.
We start by adding another interface on the switches (fa1/0/3 on SW1 and fa0/3 on SW3) to be access ports on vlan 40, and these interfaces are connected to GigabitEthernet 0/2 on the “other” ASA:
Next we move onto the firewalls and remove the IP information from the existing inside interfaces:
ASA1/act(config)# int GigabitEthernet0/1 ASA1/act(config-if)# no nameif ASA1/act(config-if)# no ip address ASA1/act(config-if)# exit
Then we can create a redundant port:
ASA1/act(config)# interface Redundant1 ASA1/act(config-if)# member-interface GigabitEthernet0/1 INFO: security-level and IP address are cleared on GigabitEthernet0/1. ASA1/act(config-if)# member-interface GigabitEthernet0/2 INFO: security-level and IP address are cleared on GigabitEthernet0/2. ASA1/act(config-if)# nameif INSIDE INFO: Security level for "INSIDE" set to 100 by default. ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10 ASA1/act(config-if)#
I could have saved a bit of typing as adding a member interface to a redundant interface will clear the security-level and IP address automatically, but its useful to do manually.
Once we have saved our config we can check on the standby ASA:
ASA1/stby# sh run : Saved <Truncated> interface GigabitEthernet0/1 no nameif no security-level no ip address ! interface GigabitEthernet0/2 no nameif no security-level no ip address ! interface Redundant1 member-interface GigabitEthernet0/1 member-interface GigabitEthernet0/2 nameif INSIDE security-level 100 ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10 !
At this point we cant ping the loopback of 10.250.1.1 on the “internet”, and to do this we need to do some NATing on the ASA, because when we clear the interface on the ASA it removes our NAT statement:
ASA1/act(config)# nat (INSIDE) 1 10.40.1.0 255.255.255.0
Once that is done we can ping the loopback again:
SW1#ping 10.250.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms SW2#ping 10.250.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms SW2#
Testing the redundant interface with failover
Now if we failover the ASAs (ASA1 being the primary, so testing from SW1) do we lose any pings?
SW1#ping 10.250.1.1 repeat 4000 Type escape sequence to abort. Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!! <omitted> !!!!! Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/2/9 ms
No we dont.
If we remove SW1 (simulating a power outage) does SW2 lose any pings?
SW2#ping 10.250.1.1 repeat 4000 Type escape sequence to abort. Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: <omitted> *Mar 5 02:12:17.681: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down <omitted> *Mar 5 02:12:18.746: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to down Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/3/34 ms
Again no ping loss. And if we turn off ASA2 (now the primary):
SW2#ping 10.250.1.1 repeat 4000 Type escape sequence to abort. Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!. *Mar 5 02:13:49.133: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down *Mar 5 02:13:50.140: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down.......!!!!!!!!!!!!!!!!!!!!!!! <omitted> Success rate is 99 percent (3992/4000), round-trip min/avg/max = 1/3/17 ms
Then yes we do lose a few pings, but only in the time it takes ASA 1 to become the active again. But losing a couple of pings is better than losing all communication! Now we can lose one switch and one ASA on different sides of the topology and still have a working (redundant) network.