CCIE Security lab: vWLC – Part 2 – Bust and Boom!

Well, it took some time, but *finally* its working. It’s been a little stressful, but perseverance and hours of googling paid off.

So the new AP arrived today. I was a little confused as it was packaged so well I thought a shower door was being delivered.

I unwrapped it and plugged it in, being the optimistic guy I am, I was hoping that it would connect to the vWLC without issue.

Yeah, so much not the case. I did try to capture as much data as I could, though, so that if/when I fixed it, I could have something useful for others. Unfortunately, my Windows laptop has a tendency to reboot spontaneously because of Windows updates, so a little was lost.

Anyway. AP gets booted up and the first thing to do is to upgrade it to a newer version. This seemed to go OK, but the AP would keep dropping off the vWLC. So I decided to try loading the recovery version instead. I had to do this via the boot console, as for some reason Flash went tits-up.

There are some Cisco docs about this, but they didn’t work, and it just sat there doing nothing. Then I googled “load_helper” and found a very helpful post that said that “ether_init” is also needed. So I rebooted the AP and tried again. This is the working command sequence:

load_helper
flash_init
format flash:
set IP_ADDR 10.1.4.53
set NETMASK 255.255.255.0
set DEFAULT_ROUTER 10.1.4.254
ether_init
tftp_init
tar -xtract tftp://10.1.4.200/c1140-rcvk9w8-tar.152-4.JA1.tar flash:
boot

This is how it looks in real life, a little slimmed down, but notice that if you mess up a command (like I did with the first tar command), you need to reenter the previous command:

ap: ether_init
ap: set IP_ADDR 10.1.4.53
ap: set NETMASK 255.255.255.0
ap: set DEFAULT_ROUTER 10.1.4.254
ap: tftp_init
ap: ether_init
ap: tar -xtract tftp://10.1.4.200/c1140_rcvk9w8_tar.152_4.JA1.tar
usage: tar   
ap: tar -xtract tftp://10.1.4.200/c1140_rcvk9w8_tar.152_4.JA1.tar flash:
Unknown cmd: tar
ap: tar ?
usage: tar   
ap: tar -xtract tftp://10.1.4.200/c1140_rcvk9w8_tar.152_4.JA1.tar flash:
Unknown cmd: tar
ap: tftp_init
ap: tar -xtract tftp://10.1.4.200/c1140_rcvk9w8_tar.152_4.JA1.tar flash:
extracting info (273 bytes)
c1140-rcvk9w8-mx/ (directory) 0 (bytes)
extracting c1140-rcvk9w8-mx/c1140-rcvk9w8-mx (6865717 bytes)........................................................ 
a lot of minutes later
extracting c1140-rcvk9w8-mx/info (273 bytes)
extracting c1140-rcvk9w8-mx/file_hashes (280 bytes)
extracting c1140-rcvk9w8-mx/final_hash (141 bytes)
extracting c1140-rcvk9w8-mx/img_sign_rel.cert (1375 bytes)
extracting c1140-rcvk9w8-mx/img_sign_rel_sha2.cert (1371 bytes)
extracting info.ver (273 bytes)
ap: boot
Loading "flash:/c1140-rcvk9w8-mx/c1140-rcvk9w8-

Once it comes back up again, we are on the new version:

Cisco IOS Software, C1140 Software (C1140-RCVK9W8-M), Version 15.2(4)JA1, RELEASE SOFTWARE (fc2)
LWAPP image version 7.5.1.73

OK, so one update is completed. Can we connect to the AP yet? Not quite, but it does start to do an update FROM the vWLC, so at least we are headed in the right direction:

 examining image...!
extracting info (282 bytes)
Image info:
    Version Suffix: k9w8-.153-3.JA
    Image Name: c1140-k9w8-mx.153-3.JA
    Version Directory: c1140-k9w8-mx.153-3.JA
    Ios Image Size: 317952
    Total Image Size: 8714752
    Image Feature: WIRELESS LAN|LWAPP
    Image Family: C1140
    Wireless Switch Management Version: 8.0.100.0
Extracting files...
c1140-k9w8-mx.153-3.JA/ (directory) 0 (bytes)
extracting c1140-k9w8-mx.153-3.JA/c1140-k9w8-xx.153-3.JA (82241        )

After a while, the AP comes back up again, and we are running another new version, courtesy of the vWLC:

POWER TABLE FILENAME = flash:/c1140-k9w8-mx.153-3.JA/T5.bin
cisco AIR-LAP1142N-E-K9 (PowerPC405ex) processor (revision B0) with 98294K/32768K bytes of memory.
Processor board ID FCZ1427W4UB
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from reload
LWAPP image version 8.0.100.0
1 Gigabit Ethernet interface
2 802.11 Radios

Notice the LWAPP image now matches the Controllers version:

(Cisco Controller) >show sysinfo 

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.100.0
RTOS Version..................................... 8.0.100.0
Bootloader Version............................... 8.0.100.0
Emergency Image Version.......................... 8.0.100.0

Build Type....................................... DATA + WPS

System Name...................................... vWLC
System Location.................................. 
System Contact................................... 
System ObjectID.................................. 1.3.6.1.4.1.9.1.1631
IP Address....................................... 10.1.4.152
IPv6 Address..................................... ::
System Up Time................................... 0 days 3 hrs 39 mins 19 secs
System Timezone Location......................... 
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... GB  - United Kingdom

--More-- or (q)uit
(Cisco Controller) >

The above output is pretty long, but the important thing is that our versions match and that the Configured Country is GB. This needs to match up with the -E- in the version details on the AP (if you are in the UK that is):

Product/Model Number                 : AIR-LAP1142N-E-K9

However, we are not out of the woods yet. The AP keeps flip-flopping about, and my frustration is growing. But we are seeing it in the vWLC, so this is still encouraging:

Cisco vWLC on UNetLab

However, it doesn’t stay stable for very long.

Some of the useful errors are below:

%DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.1.4.152:5246
%CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.1.4.152 peer_port: 5246
%CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source
%LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
%LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x4FEE418!  
%LWAPP-3-CLIENTERRORLOG: LWAPP LED Init: incorrect led state 255
%LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
%LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
%LWAPP-4-CLIENTEVENTLOG: Not sending change state post as the radio admin is down, lrad state = 5

The interesting lines are “%LWAPP-3-CLIENTERRORLOG: LWAPP LED Init: incorrect led state 255” and “DTLS_CLIENT_ERROR:…Max retransmission count reached for Connection”. There seems to be two fixes for this. The first is to disable the Self-Signed Certificate (SSC) hash validation:

(Cisco Controller) >config certificate ssc hash validation disable 

(Cisco Controller) >show certificate ssc

SSC Hash validation.............................. Disabled.

SSC Device Certificate details:

         Subject Name :
                 C=US, ST=California, L=San Jose, O=Cisco Virtual Wireless LAN Controller, 
                 CN=DEVICE-vWLC-AIR-CTVM-K9-520000020001, [email protected]

         Validity :
                 Start : 2015 Mar  3rd, 09:21:37 GMT
                 End   : 2025 Jan  9th, 09:21:37 GMT

         Hash key : e9c65b6b31a76266f28c1bddfa297780ac1bbf8a

(Cisco Controller) >

The second requirement is that the AP needs to be set as FlexConnect

Cisco vWLC on UNetLab

Once these were set, we can actually see the SSIDs!

Cisco vWLC on UNetLab
And here we can see them on the phone as well:
Cisco vWLC on UNetLab
So, this brings an end to trying to set up hardware. It now leaves the remaining tasks to be the setting up of the IPS, so that we can plug the vWLC into it and set up the ISE so that we can plug the vWSA’s APs and the IP Phone into that.
Progress is definitely being made, albeit with a bit of added frustration, but no-one ever said a CCIE would be easy.