CCIE Security Lab: FlexVPN, FYIP and the need for virtual interfaces

Flex VPN is the last VPN to set up; it’s also kind of all of the ones we have done, pushed into one. Flex VPN takes all the other VPNs, mashes them together, and this how we get Flex VPN. For a pretty good overview, check out the Cisco Live! presentation.

This one will get interesting (well it was for me, I saved most of the troubleshooting stuff, as there was just too much of it, but we’ll get to that later). The other reason it’s interesting is that we are going through three different firewalls, so need to bear this in mind.

Let’s start with GDOI-Server as the Flex-VPN client, by enabling aaa new-model and setting an authorization list:

GDOI-Server(config)#aaa new-model
GDOI-Server(config)#aaa authorization network AuthC-list local
GDOI-Server(config)#

Next, as Flex VON is all IKEv2 based, we need an IKEv2 proposal:

GDOI-Server(config)#crypto ikev2 proposal Flex-IKEv2-Proposal
IKEv2 proposal MUST either have a set of an encryption algorithm other than aes-gcm, an integrity algorithm and a DH group configured or 
 encryption algorithm aes-gcm, a prf algorithm and a DH group configured
GDOI-Server(config-ikev2-proposal)#
GDOI-Server(config-ikev2-proposal)#encryption aes-cbc-128
GDOI-Server(config-ikev2-proposal)#integrity sha512
GDOI-Server(config-ikev2-proposal)#group 5
GDOI-Server(config-ikev2-proposal)#
GDOI-Server(config-ikev2-proposal)#exit
GDOI-Server(config)#

A policy to match our proposal:

GDOI-Server(config)#crypto ikev2 policy Flex-IKEv2-Policy 
IKEv2 policy MUST have atleast one complete proposal attached 
GDOI-Server(config-ikev2-policy)#
GDOI-Server(config-ikev2-policy)#proposal Flex-IKEv2-Proposal
GDOI-Server(config-ikev2-policy)#exit
GDOI-Server(config)#

Now we need an authorization policy:

GDOI-Server(config)#crypto ikev2 authorization policy default
%Warning: This will Modify Default IKEv2 Authorization Policy. Exit if you don't want
GDOI-Server(config-ikev2-author-policy)#route set interface
GDOI-Server(config-ikev2-author-policy)#
GDOI-Server(config-ikev2-author-policy)#exit
GDOI-Server(config)#

We now create an IKEv2 profile, matching the remote address, and specifying how we are going to connect (pre-shared-key), and defining the keyring:

GDOI-Server(config)#crypto ikev2 profile Flex-IKEv2-Profile
IKEv2 profile MUST have:
   1. A local and a remote authentication method.
   2. A match identity or a match certificate or match any statement.
GDOI-Server(config-ikev2-profile)#match identity remote address 2.2.2.2
GDOI-Server(config-ikev2-profile)#authentication remote pre-share
GDOI-Server(config-ikev2-profile)#authentication local pre-share
GDOI-Server(config-ikev2-profile)#keyring local Flex-Keyring
% Invalid keyring Flex-Keyring
GDOI-Server(config-ikev2-profile)#aaa authorization group psk list AuthC-list default  
GDOI-Server(config-ikev2-profile)#
GDOI-Server(config-ikev2-profile)#exit
GDOI-Server(config)#

We create the keyring:

GDOI-Server(config)#crypto ikev2 keyring Flex-Keyring
GDOI-Server(config-ikev2-keyring)#peer Flex-CA
GDOI-Server(config-ikev2-keyring-peer)#address 2.2.2.2
GDOI-Server(config-ikev2-keyring-peer)#identity address 7.7.7.7
GDOI-Server(config-ikev2-keyring-peer)#pre-shared-key cisco
GDOI-Server(config-ikev2-keyring-peer)#
GDOI-Server(config-ikev2-keyring-peer)#exit

And add the keyring to the profile:

GDOI-Server(config-ikev2-keyring)#crypto ikev2 profile Flex-IKEv2-Profile
GDOI-Server(config-ikev2-profile)#keyring local Flex-Keyring
GDOI-Server(config-ikev2-profile)#exit
GDOI-Server(config)#

Now we create our client specifying the peer and the interface we are going to connect through:

GDOI-Server(config)#crypto ikev2 client flexvpn Flex-Client
GDOI-Server(config-ikev2-flexvpn)#
GDOI-Server(config-ikev2-flexvpn)#peer 1 2.2.2.2
GDOI-Server(config-ikev2-flexvpn)#            
GDOI-Server(config-ikev2-flexvpn)#client connect tunnel 0
                                                        ^
% Invalid input detected at '^' marker.

GDOI-Server(config-ikev2-flexvpn)#exit
GDOI-Server(config)#

Our tunnel interface does not exist yet, so let’s create it and add it to the client settings:

GDOI-Server(config)#int tun 0
GDOI-Server(config-if)#ip address negotiated
GDOI-Server(config-if)#tunnel source loopback 0
GDOI-Server(config-if)#tunnel mode ipsec ipv4
GDOI-Server(config-if)#tunnel destination 2.2.2.2
GDOI-Server(config-if)#tunnel protection ipsec profile Flex-IPSec-Profile
Profile Flex-IPSec-Profile is not defined.
GDOI-Server(config-if)#
GDOI-Server(config-if)#exit
GDOI-Server(config)#
GDOI-Server(config)#crypto ikev2 client flexvpn Flex-Client
GDOI-Server(config-ikev2-flexvpn)#client connect tunnel 0
GDOI-Server(config-ikev2-flexvpn)#
GDOI-Server(config-ikev2-flexvpn)#exit
GDOI-Server(config)#

We’ll be needing a transform set

GDOI-Server(config)#crypto ipsec transform-set Flex-TS esp-aes 256 esp-sha512-hmac     
GDOI-Server(cfg-crypto-trans)#mode tunnel
GDOI-Server(cfg-crypto-trans)#exit
GDOI-Server(config)#

And an ipsec profile:

GDOI-Server(config)#crypto ipsec profile Flex-IPSec-Profile
GDOI-Server(ipsec-profile)#set transform-set Flex-TS
GDOI-Server(ipsec-profile)#set ikev2-profile Flex-IKEv2-Profile
GDOI-Server(ipsec-profile)#exit
GDOI-Server(config)#

We attach the profile to the tunnel interface:

GDOI-Server(config)#int tun 0
GDOI-Server(config-if)#tunnel protection ipsec profile Flex-IPSec-Profile
GDOI-Server(config-if)#
*Jul 11: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
GDOI-Server(config-if)#

Let’s move on to CA-Flex (the FlexVPN server). We start by making sure that the Flex VPN tunnel source (lo0) is reachable:

CA-Flex#sh run | s router
router ospf 1
 router-id 2.2.2.2
 area 0 authentication message-digest
 network 10.0.0.0 0.255.255.255 area 0
 network 80.2.10.1 0.0.0.0 area 0
CA-Flex#conf t
CA-Flex(config)#router ospf 1
CA-Flex(config-router)#network 2.2.2.2 0.0.0.0 a 0
CA-Flex(config-router)#exit

Enable AAA new-model and create the authorization list:

CA-Flex(config-router)#aaa new-model
CA-Flex(config)#aaa authorization network AuthC-list local
CA-Flex(config)#

Create the IKEv2 authorization policy, which contains a pool for client addressing, and an ACL, which contains any routes we want to have installed as static routes on the client. These routes get important later on:

CA-Flex(config)#crypto ikev2 authorization policy default
%Warning: This will Modify Default IKEv2 Authorization Policy. Exit if you don't want
CA-Flex(config-ikev2-author-policy)#pool Flex-Pool
CA-Flex(config-ikev2-author-policy)#route set interface
CA-Flex(config-ikev2-author-policy)#route set access-list Flex-ACL           
CA-Flex(config-ikev2-author-policy)#exit
CA-Flex(config)#

Next, we have the proposal:

CA-Flex(config)#crypto ikev2 proposal Flex-IKEv2-Proposal 
IKEv2 proposal MUST either have a set of an encryption algorithm other than aes-gcm, an integrity algorithm and a DH group configured or 
 encryption algorithm aes-gcm, a prf algorithm and a DH group configured
CA-Flex(config-ikev2-proposal)# encryption aes-cbc-128
CA-Flex(config-ikev2-proposal)# integrity sha512
CA-Flex(config-ikev2-proposal)# group 5
CA-Flex(config-ikev2-proposal)#
CA-Flex(config-ikev2-proposal)#exit
CA-Flex(config)#

The policy:

CA-Flex(config)#crypto ikev2 policy Flex-IKEv2-Policy 
IKEv2 policy MUST have atleast one complete proposal attached 
CA-Flex(config-ikev2-policy)# proposal Flex-IKEv2-Proposal
CA-Flex(config-ikev2-policy)#
CA-Flex(config-ikev2-policy)#exit
CA-Flex(config)#

The keyring:

CA-Flex(config)#crypto ikev2 keyring Flex-Keyring
CA-Flex(config-ikev2-keyring)#peer GDOI-Server
CA-Flex(config-ikev2-keyring-peer)#address 7.7.7.7
CA-Flex(config-ikev2-keyring-peer)#identity address 2.2.2.2
CA-Flex(config-ikev2-keyring-peer)#pre-shared-key local cisco
CA-Flex(config-ikev2-keyring-peer)#pre-shared-key remote cisco
CA-Flex(config-ikev2-keyring-peer)#
CA-Flex(config-ikev2-keyring-peer)#exit
CA-Flex(config-ikev2-keyring)#exit
CA-Flex(config)#

The profile:

CA-Flex(config)#crypto ikev2 profile Flex-IKEv2-Profile
IKEv2 profile MUST have:
   1. A local and a remote authentication method.
   2. A match identity or a match certificate or match any statement.
CA-Flex(config-ikev2-profile)#identity remote address 7.7.7.7 255.255.255.255    
CA-Flex(config-ikev2-profile)#authentication remote pre-share
CA-Flex(config-ikev2-profile)# authentication local pre-share
CA-Flex(config-ikev2-profile)# keyring local Flex-Keyring
CA-Flex(config-ikev2-profile)#aaa authorization group psk list AuthC-list default     
CA-Flex(config-ikev2-profile)#
CA-Flex(config-ikev2-profile)#virtual-template 1
CA-Flex(config-ikev2-profile)#
CA-Flex(config-ikev2-profile)#exit

The transform set:

CA-Flex(config)#crypto ipsec transform-set Flex-TS esp-aes 256 esp-sha512-hmac         
CA-Flex(cfg-crypto-trans)# mode tunnel
CA-Flex(cfg-crypto-trans)#exit
CA-Flex(config)#

The ipsec profile:

CA-Flex(config)#crypto ipsec profile Flex-IPSec-Profile
CA-Flex(ipsec-profile)# set transform-set Flex-TS 
CA-Flex(ipsec-profile)# set ikev2-profile Flex-IKEv2-Profile
CA-Flex(ipsec-profile)#
CA-Flex(ipsec-profile)#exit
CA-Flex(config)#

We tie this to a virtual template:

CA-Flex(config)#int virtual-template 1 type tunnel
CA-Flex(config-if)#ip unnumbered lo0
CA-Flex(config-if)#tun so lo0
CA-Flex(config-if)#tun mode ipsec ipv4
CA-Flex(config-if)#tunnel protection ipsec profile Flex-IPSec-Profile
CA-Flex(config-if)#
Jul 11: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
CA-Flex(config-if)#

Then we create an IP addressing pool:

CA-Flex(config-if)#ip local pool Flex-Pool 192.168.30.10 192.168.30.20
CA-Flex(config)#

And the ACL for routes we want to install on the client. Here I am using the same as the pool.

CA-Flex(config)#ip access-list standard Flex-ACL
CA-Flex(config-std-nacl)#permit 192.168.30.0 0.0.0.255
CA-Flex(config-std-nacl)#exit
CA-Flex(config)#

That’s it for the client and the server, but at the moment, though, the two router cannot talk to each other, despite the routes being in place:

CA-Flex#sh ip route 7.7.7.7
Routing entry for 7.7.7.7/32
  Known via "ospf 1", distance 110, metric 17, type inter area
  Last update from 10.1.3.3 on GigabitEthernet0/3, 01:16:21 ago
  Routing Descriptor Blocks:
  * 10.1.3.3, from 10.1.26.254, 01:16:21 ago, via GigabitEthernet0/3
      Route metric is 17, traffic share count is 1
CA-Flex#ping 7.7.7.7  
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
CA-Flex#ping 7.7.7.7 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2 
.....
Success rate is 0 percent (0/5)
CA-Flex#

GDOI-Server#sh ip route 2.2.2.2
Routing entry for 2.2.2.2/32
  Known via "ospf 1", distance 110, metric 17, type inter area
  Last update from 10.1.11.1 on GigabitEthernet0/1, 00:20:56 ago
  Routing Descriptor Blocks:
  * 10.1.11.1, from 10.1.26.254, 00:20:56 ago, via GigabitEthernet0/1
      Route metric is 17, traffic share count is 1
GDOI-Server#ping 2.2.2.2 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 7.7.7.7 
.....
Success rate is 0 percent (0/5)
GDOI-Server#

We have the routes; we just can’t get to them. The GDOI-Server can reach 1.1.1.1, though, so this will help us find the answer pretty quickly (by looking for an ACL that permits traffic to that address):

GDOI-Server#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 7.7.7.7 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/23/27 ms
GDOI-Server#

Let’s start with the zone-based firewall:

ZBF#sh access-list
Extended IP access list 101
    10 permit tcp host 7.7.7.7 host 6.6.6.6 eq bgp (1 match)
    20 permit icmp any any (2 matches)
Extended IP access list 102
    10 permit tcp host 6.6.6.6 host 7.7.7.7 eq bgp
    20 permit icmp any any
Extended IP access list ICMP
    10 permit icmp any any echo
    20 permit icmp any any echo-reply
    30 permit icmp any any traceroute
Extended IP access list ISAKMP_IPSEC
    10 permit udp any any eq isakmp (731 matches)
    20 permit ahp any any
    30 permit esp any any (15 matches)
    40 permit udp any any eq non500-isakmp
ZBF#

This should not be the issue (at least not for the basic ping test), as we are permitting ICMP from any to any. Let’s move on:

Transparent# sh access-list | i 1.1.1.1
access-list Out->In line 5 extended permit icmp host 1.1.1.1 any (hitcnt=9) 0x176e2568 
access-list In->Out line 5 extended permit icmp any host 1.1.1.1 (hitcnt=10) 0x0d834939 
Transparent#

Looks to be a possibility. Let’s add a couple of rules, and test:

Transparent(config)# access-list Out->In extended permit icmp host 2.2.2.2 any
Transparent(config)# access-list In->Out extended permit icmp any host 2.2.2.2
Transparent(config)# 

GDOI-Server#ping 2.2.2.2 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 7.7.7.7 
.....
Success rate is 0 percent (0/5)
GDOI-Server#

Transparent# sh access-list | i 2.2.2.2                               
access-list Out->In line 22 extended permit icmp host 2.2.2.2 any (hitcnt=0) 0x4dac05ac 
access-list In->Out line 22 extended permit icmp any host 2.2.2.2 (hitcnt=1) 0x348654b0 
Transparent#

Not yet, but we can start to see the hit counts increase. Let’s move on:

FO-ASA/C2# sh access-list | i 1.1.1.1
access-list Out->In line 1 extended permit icmp any host 1.1.1.1 (hitcnt=11) 0x5e2254bb 
access-list In->Out line 5 extended permit icmp host 1.1.1.1 any (hitcnt=4) 0xee2e4853 
FO-ASA/C2#

Ok, let’s repeat the process here as well:

FO-ASA/C2(config)# access-list Out->In extended permit icmp any host 2.2.2.2
FO-ASA/C2(config)# access-list In->Out extended permit icmp host 2.2.2.2 any
FO-ASA/C2(config)# 

GDOI-Server#ping 2.2.2.2 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 7.7.7.7 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/24/32 ms
GDOI-Server#

Sweet, we have initial reachability. Now let’s go and add the rules to permit IKE between the two hosts:

FO-ASA/C2(config)# access-list Out->In extended permit esp host 7.7.7.7 host 2.2.2.2
FO-ASA/C2(config)# access-list In->Out extended permit esp host 2.2.2.2 host 7.7.7.7
FO-ASA/C2(config)# 

Transparent(config)# access-list Out->In extended permit esp host 2.2.2.2 host 7.7.7.7
Transparent(config)# access-list In->Out extended permit esp host 7.7.7.7 host 2.2.2.2
Transparent(config)#

The tunnel is not coming up yet, so let’s debug:

GDOI-Server#debug condition ip 2.2.2.2
Condition 1 set
GDOI-Server#debug cry ikev2 
IKEv2 default debugging is on
GDOI-Server#conf t
GDOI-Server(config)#logging on
GDOI-Server(config)#loggin con
GDOI-Server(config)#
*Jul 11: IKEv2:% Getting preshared key from profile keyring Flex-Keyring
*Jul 11: IKEv2:% Matched peer block 'Flex-CA'
*Jul 11: IKEv2:Searching Policy with fvrf 0, local address 7.7.7.7
*Jul 11: IKEv2:Found Policy 'Flex-IKEv2-Policy'
*Jul 11: IKEv2:SA is already in negotiation, hence not negotiating again
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Retransmitting packet 
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 2.2.2.2:500/From 7.7.7.7:500/VRF i0:f0] 
Initiator SPI : 81E13EF510EC0163 - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST 
Payload contents: 
 SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP) 

*Jul 11: IKEv2:% Getting preshared key from profile keyring Flex-Keyring
*Jul 11: IKEv2:% Matched peer block 'Flex-CA'
*Jul 11: IKEv2:Searching Policy with fvrf 0, local address 7.7.7.7
*Jul 11: IKEv2:Found Policy 'Flex-IKEv2-Policy'
*Jul 11: IKEv2:SA is already in negotiation, hence not negotiating again
*Jul 11: IKEv2-ERROR:(SESSION ID = 1,SA ID = 1):: Maximum number of retransmissions reached
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Failed SA init exchange
*Jul 11: IKEv2-ERROR:(SESSION ID = 1,SA ID = 1):Initial exchange failed: Initial exchange failed
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Abort exchange
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Deleting SA
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 2.2.2.2:500/From 7.7.7.7:500/VRF i0:f0] 
Initiator SPI : 2D17604548F256FA - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST 
Payload contents: 
 SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)

Is this caused by NAT translation? Let’s add some rules to find out

Transparent(config)# access-list Out->In extended permit udp host 2.2.2.2 host 7.7.7.7 eq 4500
Transparent(config)# access-list In->Out extended permit udp host 7.7.7.7 host 2.2.2.2 eq 4500
Transparent(config)# access-list Out->In extended permit udp host 2.2.2.2 host 7.7.7.7 eq isakmp
Transparent(config)# access-list In->Out extended permit udp host 7.7.7.7 host 2.2.2.2 eq isakmp
Transparent(config)# end
Transparent(config)# 

FO-ASA/C2(config)# access-list Out->In extended permit udp host 7.7.7.7 host 2.2.2.2 eq 4500
FO-ASA/C2(config)# access-list In->Out extended permit udp host 2.2.2.2 host 7.7.7.7 eq 4500
FO-ASA/C2(config)#

Adding these rules gets us further but is still not perfect. Let’s check the debug logs again:

*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 2.2.2.2:500/To 7.7.7.7:500/VRF i0:f0] 
Initiator SPI : 7E3708858C9DD409 - Responder SPI : 1CA9B1A417F3D9F9 Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE 
Payload contents: 
 VID IDr AUTH CFG SA TSi TSr NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS) 

*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Process auth response notify
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Searching policy based on peer's identity '2.2.2.2' of type 'IPv4 address'
*Jul 11: IKEv2:Searching Policy with fvrf 0, local address 7.7.7.7
*Jul 11: IKEv2:Found Policy 'Flex-IKEv2-Policy'
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Verify peer's policy
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Peer's policy verified
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Get peer's authentication method
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Peer's authentication method is 'PSK'
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Get peer's preshared key for 2.2.2.2
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Verify peer's authentication data
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Use preshared key for id 2.2.2.2, key len 5
*Jul 11: IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
*Jul 11: IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Verification of peer's authenctication data PASSED
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Check for EAP exchange
*Jul 11: IKEv2:Using mlist AuthC-list and username default for group author request
*Jul 11: IKEv2:(SA ID = 1):[IKEv2 -> AAA] Authorisation request sent
*Jul 11: IKEv2:(SA ID = 1):[AAA -> IKEv2] Received AAA authorisation response
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Received valid config mode data
*Jul 11: IKEv2:Config data recieved:
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Config-type: Config-reply 
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Attrib type: ipv4-addr, length: 4, data: 192.168.30.10
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Attrib type: ipv4-subnet, length: 8, data: 2.2.2.2 255.255.255.255
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Attrib type: ipv4-subnet, length: 8, data: 192.168.30.0 255.255.255.0
*Jul 11: IKEv2:VPN Route Added 2.2.2.2 255.255.255.255 via Tunnel0 in vrf global
*Jul 11: IKEv2:VPN Route Added 192.168.30.0 255.255.255.0 via Tunnel0 in vrf global
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Set received config mode data
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Processing IKE_AUTH message
*Jul 11: IKEv2:IPSec policy validate request sent for profile Flex-IKEv2-Profile with psh index 1.
*Jul 11: IKEv2:(SA ID = 1):[IPsec -> IKEv2] Callback received for the validate proposal - PASSED.
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):IKEV2 SA created; inserting SA into database. SA lifetime timer (86400 sec) started
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Session with IKE ID PAIR (2.2.2.2, 7.7.7.7) is UP
*Jul 11: IKEv2:IKEv2 MIB tunnel started, tunnel index 1
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Load IPSEC key material
*Jul 11: IKEv2:(SA ID = 1):[IKEv2 -> IPsec] Create IPsec SA into IPsec database
*Jul 11: IKEv2:(SA ID = 1):[IPsec -> IKEv2] Creation of IPsec SA into IPsec database PASSED
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Config-type: Config-set 
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Attrib type: ipv4-subnet, length: 8, data: 192.168.30.10 255.255.255.255
*Jul 11: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
*Jul 11: %ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel0 (incomplete) - looped chain attempting to stack
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Checking for duplicate IKEv2 SA
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):No duplicate IKEv2 SA found
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Sending info exch config
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.  
Payload contents: 
 CFG
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Checking if request will fit in peer window 
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 2.2.2.2:500/From 7.7.7.7:500/VRF i0:f0] 
Initiator SPI : 7E3708858C9DD409 - Responder SPI : 1CA9B1A417F3D9F9 Message id: 2
IKEv2 INFORMATIONAL Exchange REQUEST 
Payload contents: 
 ENCR 
*Jul 11: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
*Jul 11: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Retransmitting packet 
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Retransmitting packet 
*Jul 11: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 2.2.2.2:500/From 7.7.7.7:500/VRF i0:f0] 
Initiator SPI : 7E3708858C9DD409 - Responder SPI : 1CA9B1A417F3D9F9 Message id: 2
IKEv2 INFORMATIONAL Exchange REQUEST 
Payload contents: 
 ENCR 
*Jul 11: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
Transparent#

Everything comes up, but then the tunnel drops due to recursive routing. This is caused by the server, advertising the interface route back through the tunnel. Let’s remove this route:

CA-Flex(config)#crypto ikev2 authorization policy default
CA-Flex(config-ikev2-author-policy)#no route set interface
CA-Flex(config-ikev2-author-policy)#

Flex is idle, but interface is up:

GDOI-Server#sh ip int bri
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.1.12.7       YES NVRAM  up                    up      
GigabitEthernet0/1         10.1.11.7       YES NVRAM  up                    up      
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/3         unassigned      YES NVRAM  administratively down down    
Loopback0                  7.7.7.7         YES NVRAM  up                    up      
Tunnel0                    192.168.30.11   YES NVRAM  up                    up      
GDOI-Server#sh cry ikev2 clie flex

  Profile : Flex-Client
  Current state:IDLE
  Backup group: Default
  Tunnel interface : Tunnel0
GDOI-Server#

GDOI is not well, though:

GDOI-Server#sh cry gdoi | i Group Name|Member
    Group Name               : GDOI-G1 (Unicast)
    Group Members            : 0
    Group Name               : GDOI-G2 (Unicast)
    Group Members            : 1
GDOI-Server#

GDOI-G1#sh cry gdoi
GROUP INFORMATION

    Group Name               : GDOI-G1
    Group Identity           : 1
    Group Type               : GDOI (ISAKMP)
    Crypto Path              : ipv4
    Key Management Path      : ipv4
    Rekeys received          : 0
    IPSec SA Direction       : Both

     Group Server list       : 7.7.7.7
                               
Group Member Information For Group GDOI-G1:
    IPSec SA Direction       : Both
    ACL Received From KS     : 


       allowable rekey cipher: any
       allowable rekey hash  : any
       allowable transformtag: any ESP

    Rekeys cumulative
       Total received        : 0
       After latest register : 0
       Rekey Received        : never

 ACL Downloaded From KS UNKNOWN:

TEK POLICY for the current KS-Policy ACEs Downloaded:
GDOI-G1#

This is more to do with the Easy VPN set up before, I just didn’t see it last time, and this is probably due to no checking properly. I should have saved and reloaded the routers to makes sure. Here is the issue (as I see it) in picture format:

ISAKMP profiles and virtual templates

So, I need to differentiate the following:

1: Flex VPN traffic coming into 7.7.7.7 from GET VPN traffic coming into the same address.
2: Easy VPN traffic coming into GDOI-G1’s Gi0/1 interface from GET VPN traffic on the same interface.

Interestingly, GDOI-G2 is all fine, which does beg the question as to whether the GDOI-G1 issues are all on GDOI-G1.

After doing some digging, I think I am over analyzing the issue. While a state of IDLE may not seem like what we want, it is, much like the QM_IDLE from DMVPN we looked previously. So, although I took this to mean that there was an issue, actually all is fine, and we can prove this by adding a new loopback on CA-Flex, and just advertising that to the Flex VPN client:

CA-Flex(config-router)#do sh run int Loopback 22 | b interface 
interface Loopback22
 ip address 22.22.22.22 255.255.255.255
end

CA-Flex(config-router)#do sh access-list                      
Standard IP access list Flex-ACL
    10 permit 22.22.22.22
CA-Flex(config-router)#


GDOI-Server(config)#do sh cry ikev2 cli flex

  Profile : Flex-Client
  Current state:IDLE
  Backup group: Default
  Tunnel interface : Tunnel0
GDOI-Server(config)#do sh ip route | i Tunnel
S        22.22.22.22 is directly connected, Tunnel0
C        192.168.30.11 is directly connected, Tunnel0
GDOI-Server(config)#do ping 22.22.22.22      
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 22.22.22.22, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 26/27/28 ms
GDOI-Server(config)#
GDOI-Server(config)#do ping 22.22.22.22 so tun 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 22.22.22.22, timeout is 2 seconds:
Packet sent with a source address of 192.168.30.11 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 21/30/43 ms
GDOI-Server(config)#

I just spent a long time trying to fix a problem that was not a problem at all, because I neglected to test the basics. The thing is that we cannot ping the Tunnel0 interface IP address from CA-Flex. But this is a routing issue, if I had tried pinging it from the advertised loopback, it would have worked fine:

CA-Flex(config-router)#do ping 192.168.30.11 so loo22
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.11, timeout is 2 seconds:
Packet sent with a source address of 22.22.22.22 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/29/34 ms
CA-Flex(config-router)#

So, our Flex VPN is working as it should. Let’s go and fix GDOI-G1. After much fixing (ahem, read clutching at straws, and trial and error) I got GDOI and Easy VPN working together. Here is the resulting configuration:

GDOI-G1#sh run
hostname GDOI-G1
!
username cisco password 0 ccie
!         
crypto keyring Get-Keyring  
  pre-shared-key address 7.7.7.7 key cisco
!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
!
crypto isakmp policy 20
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp key cisco address 0.0.0.0        
crypto isakmp profile GET-ISAK-Prof
   keyring Get-Keyring
   match identity address 7.7.7.7 255.255.255.255 
   virtual-template 10
!
crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac 
 mode tunnel
crypto ipsec transform-set Ez-TS esp-aes 256 esp-sha256-hmac 
 mode tunnel
!
crypto ipsec profile GET-Profile
 set transform-set GET-TS 
 set isakmp-profile GET-ISAK-Prof
!
crypto ipsec client ezvpn EZ-Group
 connect auto
 group EZ-Group key cisco
 mode client
 peer 6.6.6.6
 xauth userid mode interactive
!
crypto gdoi group GDOI-G1
 identity number 1
 server address ipv4 7.7.7.7
 client registration interface GigabitEthernet0/1
!
crypto map G1 10 gdoi 
 set group GDOI-G1
!
interface Loopback0
 ip address 8.8.8.8 255.255.255.0
 crypto ipsec client ezvpn EZ-Group inside
!
interface Loopback88
 ip address 88.88.88.88 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 10.1.13.8 255.255.255.0
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 cisco
 duplex auto
 speed auto
 media-type rj45
 crypto ipsec client ezvpn EZ-Group
!
interface Virtual-Template10 type tunnel
 ip unnumbered Loopback0
 tunnel source GigabitEthernet0/1
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile GET-Profile
 crypto map G1
!
router ospf 1
 router-id 8.8.8.8
 area 1 authentication message-digest
 network 8.8.8.8 0.0.0.0 area 1
 network 10.1.13.0 0.0.0.255 area 1
GDOI-G1#

So, before we go through the explanation, let’s show the proof:

GDOI-Server#sh cry gdoi | i Name|Member
    Group Name               : GDOI-G1 (Unicast)
    Group Members            : 1
      Profile Name           : GET-Profile
    Group Name               : GDOI-G2 (Unicast)
    Group Members            : 1
      Profile Name           : GET-Profile
GDOI-Server#

GDOI-G1#sh cry gdoi
GROUP INFORMATION

    Group Name               : GDOI-G1
    Group Identity           : 1
    Group Type               : GDOI (ISAKMP)
    Crypto Path              : ipv4
    Key Management Path      : ipv4
    Rekeys received          : 1
    IPSec SA Direction       : Both

     Group Server list       : 7.7.7.7
                               
Group Member Information For Group GDOI-G1:
    IPSec SA Direction       : Both
    ACL Received From KS     : gdoi_group_GDOI-G1_temp_acl

    Group member             : 10.1.13.8       vrf: None
       Local addr/port       : 10.1.13.8/848
       Remote addr/port      : 7.7.7.7/848
       fvrf/ivrf             : None/None
       Version               : 1.0.17
       Registration status   : Registered
       Registered with       : 7.7.7.7
       Re-registers in       : 3178 sec
       Succeeded registration: 1
       Attempted registration: 1
       Last rekey from       : 7.7.7.7
       Last rekey seq num    : 13
       Unicast rekey received: 1
       Rekey ACKs sent       : 1
       Rekey Rcvd(hh:mm:ss)  : 00:03:24
       DP Error Monitoring   : OFF
       IPSEC init reg executed    : 0
       IPSEC init reg postponed   : 0
       Active TEK Number     : 2
       SA Track (OID/status) : disabled

       allowable rekey cipher: any
       allowable rekey hash  : any
       allowable transformtag: any ESP

    Rekeys cumulative
       Total received        : 1
       After latest register : 1
       Rekey Acks sents      : 1

 ACL Downloaded From KS 7.7.7.7:
   access-list   permit ip 192.168.2.0 0.0.0.255 192.168.2.0 0.0.0.255

KEK POLICY:
    Rekey Transport Type     : Unicast
    Lifetime (secs)          : 63689
    Encrypt Algorithm        : AES
    Key Size                 : 256     
    Sig Hash Algorithm       : HMAC_AUTH_SHA
    Sig Key Length (bits)    : 1296    

TEK POLICY for the current KS-Policy ACEs Downloaded:
  Virtual-Template10:
    IPsec SA:
        spi: 0xA9EF50A8(2851033256)
        KGS: Disabled
        transform: esp-3des esp-sha-hmac 
        sa timing:remaining key lifetime (sec): (3396)
        Anti-Replay(Counter Based) : 64
        tag method : disabled
        alg key size: 24 (bytes)
        sig key size: 20 (bytes)
        encaps: ENCAPS_TUNNEL
          
    IPsec SA:
        spi: 0xFC564D7D(4233514365)
        KGS: Disabled
        transform: esp-3des esp-sha-hmac 
        sa timing:remaining key lifetime (sec): (181)
        Anti-Replay(Counter Based) : 64
        tag method : disabled
        alg key size: 24 (bytes)
        sig key size: 20 (bytes)
        encaps: ENCAPS_TUNNEL

GDOI-G1#
GDOI-G1# sh access-list
GDOI-G1#sh ip int bri | e unas
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/1         10.1.13.8       YES NVRAM  up                    up      
Loopback0                  8.8.8.8         YES NVRAM  up                    up      
Loopback88                 88.88.88.88     YES NVRAM  up                    up      
Loopback10000              192.168.10.18   YES TFTP   up                    up      
NVI0                       10.1.13.8       YES unset  up                    up      
Virtual-Template10         8.8.8.8         YES unset  up                    down    

GDOI-G1#

EzVPN-Server#ping 192.168.10.18
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.18, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 13/16/20 ms
EzVPN-Server#

So, what was changed, and why?

In the configuration, the GET VPN interface has been moved to a virtual interface. This allows us to still use the same outside interface (Gi0/1) for Easy VPN and GET VPN, but the two can live happily alongside each other. Here is a side-by-side comparison of GDOI-G1 and GDOI-G2 so we can see the differences:

GDOI-G1 GDOI-G2
Crypto Keyring crypto keyring Get-Keyring
pre-shared-key address 7.7.7.7 key cisco
** Not required **
ISAKMP Policy crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
crypto isakmp key cisco address 0.0.0.0
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
crypto isakmp key cisco address 0.0.0.0
ISAKMP Profile crypto isakmp profile GET-ISAK-Prof
keyring Get-Keyring
match identity address 7.7.7.7 255.255.255.255
virtual-template 10
** Not Required **
Transform Set crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac
mode tunnel
crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac
mode tunnel
IPSec Profile crypto ipsec profile GET-Profile
set transform-set GET-TS
set isakmp-profile GET-ISAK-Prof
crypto ipsec profile GET-Profile
set transform-set GET-TS
GDOI Group crypto gdoi group GDOI-G1
identity number 1
server address ipv4 7.7.7.7
client registration interface GigabitEthernet0/1
crypto gdoi group GDOI-G2
identity number 2
server address ipv4 7.7.7.7
client registration interface GigabitEthernet0/2
Crypto Map crypto map G1 10 gdoi
set group GDOI-G1
crypto map G2 10 gdoi
set group GDOI-G2
Outside interface interface GigabitEthernet0/1
ip address 10.1.13.8 255.255.255.0
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 cisco
duplex auto
speed auto
media-type rj45
crypto ipsec client ezvpn EZ-Group
interface GigabitEthernet0/2
ip address 10.1.14.9 255.255.255.0
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 cisco
duplex auto
speed auto
media-type rj45
crypto map G2
Virtual Template interface Virtual-Template10 type tunnel
ip unnumbered Loopback0
tunnel source GigabitEthernet0/1
tunnel mode ipsec ipv4
tunnel protection ipsec profile GET-Profile
crypto map G1
** Not required **

As you can see, we push the pre-shared-key into a Keyring and set up an ISAKMP profile, to use this keyring, match it to the tunnel destination (GDOI-Server’s loopback address), and point it to a virtual template. Under the IPSec profile, we attach the ISAKMP profile, but then unlike GDOI-G2, we do not call the GDOI crypto map on the external interface (Gig0/1), instead, this is called by the virtual template. This virtual template also uses the IPSec profile, the IP address from loopback0, and has the source interface as Gig0/1.

Now, let’s compare how Easy VPN was configured before, and how it is now:

Easy-VPN initial Easy VPN now
ISAKMP policy: crypto isakmp policy 20
encr aes 256
authentication pre-share
group 2
crypto isakmp policy 20
encr aes 256
authentication pre-share
group 2
Easy VPN Client config: crypto ipsec client ezvpn
connect auto
group EZ-Group key cisco
mode client
peer 6.6.6.6
crypto ipsec client ezvpn EZ-Group
connect auto
group EZ-Group key cisco
mode client
peer 6.6.6.6
xauth userid mode interactive
Outside interface: int gi0/1
crypto ipsec client ez EZ-Group
interface GigabitEthernet0/1
crypto ipsec client ezvpn EZ-Group
Inside interface: int lo0
crypto ipsec client ez EZ-Group inside
interface Loopback0
crypto ipsec client ezvpn EZ-Group inside

Barring the xauth, no changes. I only needed to change one or the other, but (as I found), not both VPNs. This took me some time to figure out. While we do need the separation between the two VPNs, and can achieve this through the keyring, ISAKMP profile, and virtual-template, we only need to do this once if we have two VPNs, when we looked at ISAKMP profiles before, we had three VPNs and needed two ISAKMP profiles. So, whilst we can have one “default”, additional VPNs will require separation through ISAKMP profiles and virtual interfaces.

That’s pretty much it for this topology. I could go and extend out EIGRP across the VPNs and implement the odd bit of security here and there, but I am itching to upgrade UNetLab to the new version, and start a new lab!

Oh, and if you were wondering what “FYIP” means – it stands for Fuck You ISAKMP Profiles.