Cisco ASA failover, redundant interfaces, Catalyst HSRP and power

Most networks have redundancy (or at least should do). This can be in the form of additional cables cross-connecting devices, dual power supplies going to different feeds and HSRP on switches. All of this is good stuff, but some things can easily be taken for granted.

Consider the following topology:

We have two ASAs in a failover pair (Outside interface 10.30.1.1 with standby ip 10.30.1.2), both connecting to a switch (10.30.1.254), that has a loopback interface assigned to it to simulate the internet (10.250.1.1). The ASAs also connect to a pair of switches, set up in an HSRP group (ASA inside interface 10.40.1.1, standby ip 10.40.1.10, switch vlan 40 ip 10.40.1.254). The topology looks like this:

It looks pretty redundant right? But let’s work through the config and see what could go wrong. We’ll start by setting up the ASAs in a failover pair.

Cisco ASA failover setup

On the first ASA we set up failover as follows:

ciscoasa(config)# enable password cisco
ciscoasa(config)# hostname ASA1
ASA1(config)# failover
ASA1(config)# failover lan unit primary
ASA1(config)# failover lan interface FAILOVER GigabitEthernet0/3
INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces
ASA1(config)# failover key fail0ver
ASA1(config)# failover replication http
ASA1(config)# failover link FAILOVER GigabitEthernet0/3
ASA1(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.1
ASA1(config)# int gi0/3
ASA1(config-if)# no shut
ASA1(config-if)# exit
ASA1(config)# exit
ASA1# copy run start

The second ASA is similar:

ciscoasa(config)# enable password cisco
ciscoasa(config)# hostname ASA2
ASA2(config)# failover
ASA2(config)# failover lan unit secondary
ASA2(config)# failover lan interface FAILOVER GigabitEthernet0/3
INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces
ASA2(config)# failover key fail0ver
ASA2(config)# failover replication http
ASA2(config)# failover link FAILOVER GigabitEthernet0/3
ASA2(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.2
ASA2(config)# int gi0/3
ASA2(config-if)# no shut
ASA2(config-if)#exit
ASA2(config)#exit
ASA2# copy run start

We can then save our configuration, so that the partner (or mate) firewall gets sent the active configuration:

ASA1# copy run start
ASA1# Beginning configuration replication: Sending to mate.
End Configuration Replication to mate

I do like to set the prompt to show me easily which firewall I am currently on:

ASA1#
ASA1# conf t
ASA1(config)# prompt hostname state
ASA1/act(config)# exit
ASA1/act# copy run start

Source filename [running-config]?
Cryptochecksum: 9f0c4260 95438d98 2d1439ef 0704bf77

2693 bytes copied in 3.310 secs (897 bytes/sec)
ASA1/act#

Our partner ASA will now change to show its status in the prompt:

ASA1#
ASA1/stby# Cryptochecksum: 0df25109 9f4fecb5 cebb9db5 93664d73

ASA1/stby#
2691 bytes copied in 3.230 secs (897 bytes/sec)
ASA1/stby#

Now we can configure our connection to the “internet” and to the inside network:

ASA Outside (internet) connection

ASA1/act# conf t
ASA1/act(config)# int Gi0/0
ASA1/act(config-if)# nameif OUTSIDE
INFO: Security level for "OUTSIDE" set to 0 by default.
ASA1/act(config-if)# ip address 10.30.1.1 255.255.255.0 standby 10.30.1.2
ASA1/act(config-if)# no shut
ASA1/act(config-if)#exit
ASA1/act(config)# exit
ASA1/act# copy run start

ASA Inside connection

ASA1/act(config-if)# int gi 0/1
ASA1/act(config-if)# nameif INSIDE
INFO: Security level for "INSIDE" set to 100 by default.
ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
ASA1/act(config-if)# no shut
ASA1/act(config-if)# exit
ASA1/act(config)# exit
ASA1/act# copy run start
Now we have connections on the inside and outside:

Switch HSRP setup

I have covered HSRP a few times now, so won’t go into any details, but here is the configuration walkthrough:
Switch(config)#hostname SW1
SW1(config)#vlan 40
SW1(config-vlan)#exit
SW1(config)#int vlan 40
SW1(config-if)#ip address 10.40.1.2 255.255.255.0
00:06:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down
SW1(config-if)#standby 40 ip 10.40.1.254
SW1(config-if)#standby 40 pri 150
SW1(config-if)#
00:09:16: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down
SW1(config-if)#exit
00:09:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to up
SW1(config)#int fastEthernet 1/0/2
SW1(config-if)#switchport trunk encap dot1q
SW1(config-if)#switchport mode trunk
00:09:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up
SW1(config-if)#
00:10:10: %HSRP-6-STATECHANGE: Vlan40 Grp 40 state Standby -> Active
SW1(config-if)#

Switch(config)#hostname SW2
SW2(config)#vlan 40
SW2(config-vlan)#exit
SW2(config)#int vlan 40
*Mar  1 00:06:56.997: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down
SW2(config-if)#ip address 10.40.1.3 255.255.255.0
SW2(config-if)#standby 40 ip 10.40.1.254
SW2(config-if)#standby pri 90
SW2(config-if)#exit
SW2(config)#int fa0/2
SW2(config-if)#switchport trunk encap dot1q
SW2(config-if)#switchport mode trunk
SW2(config-if)#
*Mar  1 00:09:22.447: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
*Mar  1 00:09:25.517: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up
*Mar  1 00:09:55.515: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up
*Mar  1 00:10:36.561: %HSRP-5-STATECHANGE: Vlan40 Grp 40 state Speak -> Standby
SW2(config-if)#

Switch to ASA configuration

SW1(config)#int fa1/0/1
SW1(config-if)#switchport mode access
SW1(config-if)#switchport access vlan 40
SW1(config-if)#exit
SW1(config)#exit
SW1#ping 10.40.1.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms

SW2(config)#int fa0/1
SW2(config-if)#switchport mode access
SW2(config-if)#switchport access vlan 40
SW2(config-if)#exit
SW2(config)#exit
SW2#
*Mar  1 01:09:09.651: %SYS-5-CONFIG_I: Configured from console by console
SW2#ping 10.40.1.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

ASAs to “Internet” Configuration

On the active ASA we need to set a route out to the internet:

ASA1/act# sh run | i route
route OUTSIDE 0.0.0.0 0.0.0.0 10.30.1.254 1

Internet configuration

The internet is very simple, we have a loopback that will be our end-point for testing, and a vlan for the ASAs to communicate with:
3550-SW3#sh run
interface Loopback0
  ip address 10.250.1.1 255.255.255.0
!
interface FastEthernet0/1
  switchport access vlan 10
  switchport mode access
!
interface FastEthernet0/11
  switchport access vlan 10
  switchport mode access
!
interface Vlan10
  ip address 10.30.1.254 255.255.255.0

NAT statements to allow access

To allow the inside to talk to the internet we need to set up NATing on the ASAs:
global (OUTSIDE) 1 interface
nat (INSIDE) 1 10.40.1.0 255.255.255.0

Testing

Let’s see how out connectivity looks:
ASA1/act# ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

ASA1/stby> ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

SW1#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW2#
So far so good!

Forcing failover and simulating outages 

We can simulate issues in a number of ways. I started by removing the power to SW1. This made SW2 go active for vlan 40. SW2 has a good connection to ASA2 so pings were 100% successful.
 
With ASA1 as the standby we can force a failover from ASA2:
ASA1/stby# failover active

    Switching to Active

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
....!
Success rate is 20 percent (1/5), round-trip min/avg/max = 1/1/1 ms

ASA1/act# Waiting for the earlier webvpn instance to terminate...
Previous instance shut down. Starting a new one.

    Switching to Standby
At this point the firewalls failed back, due to the traffic generated by SW2, so that ASA2 became the active mate again. If ASA2 had the power removed then they would not be able to fail back, and therefore SW2 would have no way to route the traffic over to SW1 (which would be the active firewall), and therefore the inside of the network would lose all internet communication.
 
So we can survive one ASA power outage and one switch outage (ASA1 and SW1 or ASA2 and SW2) but not if we lose SW2 and ASA1 or SW1 and ASA2.
 
So how can we get better resiliency? We can do this with a cable from SW1 to ASA2 and from SW2 to ASA1. This does require some reconfiguration on the ASAs, but how can we do this? We certainly can’t have another physical interface on the ASAs with the same IP range, as we would get an error informing us of an IP overlap. They key work here is physical interface. We can however set up a different type of interface to achieve this.

Cisco ASA Redundant interfaces

Redundant interfaces are two or more interfaces joined together in a redundancy group. They act like physical interfaces, in that they have an IP address, but with the added bonus of having multiple links.
 
We start by adding another interface on the switches (fa1/0/3 on SW1 and fa0/3 on SW3) to be access ports on vlan 40, and these interfaces are connected to GigabitEthernet 0/2 on the “other” ASA:
 
 
 
Next we move onto the firewalls and remove the IP information from the existing inside interfaces:
ASA1/act(config)# int GigabitEthernet0/1
ASA1/act(config-if)# no nameif
ASA1/act(config-if)# no ip address
ASA1/act(config-if)# exit
Then we can create a redundant port:
ASA1/act(config)# interface Redundant1
ASA1/act(config-if)# member-interface GigabitEthernet0/1
INFO: security-level and IP address are cleared on GigabitEthernet0/1.
ASA1/act(config-if)# member-interface GigabitEthernet0/2
INFO: security-level and IP address are cleared on GigabitEthernet0/2.
ASA1/act(config-if)# nameif INSIDE
INFO: Security level for "INSIDE" set to 100 by default.
ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
ASA1/act(config-if)#
I could have saved a bit of typing as adding a member interface to a redundant interface will clear the security-level and IP address automatically, but its useful to do manually.
 
Once we have saved our config we can check on the standby ASA:
ASA1/stby# sh run
: Saved
<Truncated>
interface GigabitEthernet0/1
  no nameif
  no security-level
  no ip address
!
interface GigabitEthernet0/2
  no nameif
  no security-level
  no ip address
!
interface Redundant1
  member-interface GigabitEthernet0/1
  member-interface GigabitEthernet0/2
  nameif INSIDE
  security-level 100
  ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
!
At this point we cant ping the loopback of 10.250.1.1 on the “internet”, and to do this we need to do some NATing on the ASA, because when we clear the interface on the ASA it removes our NAT statement:
ASA1/act(config)# nat (INSIDE) 1 10.40.1.0 255.255.255.0
Once that is done we can ping the loopback again:
SW1#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW2#

Testing the redundant interface with failover

Now if we failover the ASAs (ASA1 being the primary, so testing from SW1) do we lose any pings?
SW1#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
<omitted>
!!!!!
Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/2/9 ms
No we dont.
 
If we remove SW1 (simulating a power outage) does SW2 lose any pings?
SW2#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
<omitted>
*Mar  5 02:12:17.681: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
<omitted>
*Mar  5 02:12:18.746: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to down
Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/3/34 ms
Again no ping loss. And if we turn off ASA2 (now the primary):
SW2#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
*Mar  5 02:13:49.133: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar  5 02:13:50.140: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down.......!!!!!!!!!!!!!!!!!!!!!!!
<omitted>
Success rate is 99 percent (3992/4000), round-trip min/avg/max = 1/3/17 ms
Then yes we do lose a few pings, but only in the time it takes ASA 1 to become the active again. But losing a couple of pings is better than losing all communication! Now we can lose one switch and one ASA on different sides of the topology and still have a working (redundant) network.

25 Comments

  1. Anonymous April 17, 2015
  2. Anonymous April 23, 2015
  3. Sachin Babar June 8, 2015
  4. Stuart Fordham June 10, 2015
  5. Ratha Chum December 17, 2015
  6. Anonymous January 27, 2016
  7. Ratha Chum March 22, 2016
  8. Stuart Fordham March 22, 2016
  9. Ahmadov February 19, 2017
    • Stuart Fordham March 4, 2017
  10. Me March 18, 2017
  11. Alejandro June 22, 2017
    • Stuart Fordham June 22, 2017
  12. Akb September 27, 2017
  13. Endstep July 5, 2018
  14. Zamil March 18, 2019
    • Stuart Fordham March 18, 2019
      • Zamil March 18, 2019
  15. Arnaud November 28, 2019
  16. Neil January 2, 2022
    • Stuart Fordham January 2, 2022
  17. Neil January 3, 2022

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.