Friday, September 13, 2013

ACS 5.x: Using Identity Groups to Simplify Authorization Policies

One of the things that I rarely see implemented in ACS 5.x are Identity Groups.  This is an incredibly powerful tool that can be implemented in almost every ACS installation.  It's one of those things that is really useful when dealing with failover, multiple identity stores, etc.

I was reminded how powerful this is when I received a call from a customer who's entire wireless and VPN environment was down due to issues with their ACS box.  The root cause?  All of the domain controllers were having issues.  Sadly, LDAP was still functioning, but they had never tested the LDAP integration and it didn't work.

For those that do implement multiple identity stores, what I usually see is a HUGE list of Authorization rules for each combination of Group and Store.  This gets really nasty when trying to troubleshoot why you are getting a particular undesired result.

Here I will be showing how to authenticate against multiple identity stores in order to deal with the failure of AD, LDAP and finally roll to internal users.

Thursday, September 5, 2013

Using Fiddler2 to troubleshoot Captive Web Portals

I find in wireless a lot of people aren't familiar with Fiddler.  I was introduced to Fiddler when I was writing web services and needed to diagnose why my web services weren't working.  Fiddler is a web proxy for windows that runs under the .Net framework. (Sorry Mac users).  I think to think of Fiddler as Wireshark for Web Services.   While you can proxy for other devices, that's beyond the scope of this article.

One of the things I have the most difficult with is troubleshooting Captive Web Portals.  There are a lot of things that can go wrong.  Between DHCP, DNS and Web Intercept, external web servers, etc, it can be especially difficult to troubleshoot to narrow down where the issue is.

Layer on top of this things like on-boarding product or RADIUS Change of Authorization, Cloud services and content filtering, things can become very complicated in a hurry.  While Fiddler doesn't make it any simpler, it does give us the visibility to the web interactions between the client and server. 

I'll assume that you've gone over to and downloaded/installed the software.  It's free.  I would suggest installing it on a windows laptop with a fresh install.  Why?  Just because you won't have a million apps trying to make calls to cloud services while you're troubleshooting.

So I launch fiddler on my broken Web Auth wireless network and try to go to  Care to guess why this isn't working?

Hmm,  DNS Lookup Failed.  Although, since I broke the DNS server for this example, I know this is why and after the DNS issue is resolved, this is what we get.

Looking at the response you can see there is a Web Auth Redirection and the the Virtual IP of the WLC.  Prettying this up we see:

          <TITLE> Web Authentication Redirect</TITLE>
          <META http-equiv="Cache-control" content="no-cache">
          <META http-equiv="Pragma" content="no-cache"><META http-equiv="Expires" content="-1">
          <META http-equiv="refresh" content="1; URL=">

If you click on the subsequent entries, you can look at both the client request and response.  If you are doing HTTPS, you can even enable HTTPS decryption so you can see what is going into the tunnel, which is something that's a lot harder to do with low level tools like Wireshark.

Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.

Secure Protocol: Tls
Cipher: Aes256 256bits
Hash Algorithm: Sha1 160bits
Key Exchange: RsaKeyX 1024bits

== Server Certificate ==========
  CN=, OU=DeviceSSL (WebAuth), O=Cisco Systems Inc., C=US

  CN=, OU=DeviceSSL (WebAuth), O=Cisco Systems Inc., C=US

[Serial Number]

[Not Before]
  7/8/2013 6:00:01 PM

[Not After]
  7/8/2023 6:00:01 PM


With SSL Decryption in place, you can actually look and see that it didn't work this time because my username and password were not correct.  Now be aware that to do this, i have to accept the fiddler SSL cert and approve each tunnel to decrypt, so for a hacking/sniffing tool, it's not very useful.

Now while Fiddler is a good tool to have in your arsenal, it won't replace packet capture software for other troubleshooting.  You also might find uses for this outside of the wireless world.  My coworker uses this for testing with load-balancers and other higher layer services.  So happy Fiddling and hopefully you find this tool useful.

Monday, September 2, 2013

Begun, the Chipset Wars Have

As you might expect, 802.11ac was a hot topic at Wireless Field Day 5.  While there wasn't a lot of talk about it onscreen, there was a lot of talk about chipset vendors between the delegates and sponsors.  Surprisingly, Meru  and Xirrus were some of the most open about talking about their chipset vendors of choice.  Props to Xirrus for being the only vendor to admit they were using Qualcomm Atheros on-air for a yet-to-ship product.

The general consensus was that Broadcom surprised a lot of people with their focus on the enterprise and their speed to market.  At the time of writing this, all of the shipping enterprise 802.11ac access points are Broadcom chipsets in the enterprise space (Cisco, Aruba and Meru).

Atheros was the king of the enterprise AP market in 802.11n.  With the exception of Cisco who is known for their Marvel chipsets, almost every enterprise class AP vendor used Atheros for their 11n chipsets.  Now Broadcom is the first to market and a number of the enterprise manufacturers have made the jump to Broadcom due to a significant lead time of getting their gear to market.

But being early to market didn't come without a price.  There are some restrictions around the number of encrypted clients, number of beam-formed clients and whether they support promiscuous  packet capture.

I would expect to see vendors bringing their Atheros based chipsets to market soon. Xirrus passed around their 802.11ac module based on Qualcomm Atheros.  I would expect to see a number of product announcement in the coming months and for the vendors to promote how they are different from their competitors products.

The real question is whether vendors who have jumped on the Broadcom bandwagon will continue to stay there, or if they will make the jump back to Atheros when those products become available.  This will also help AP vendors help differentiate themselves from these early generation products.  One thing you can bet on, it will be an exciting ride.

My thoughts:

Competition in this space will be good for everyone.  It will cause chipset manufactures to innovate with features and functionality, drive down price and generate product differentiation for enterprise customers.

Tuesday, August 27, 2013

Getting On Board with On-boarding products

The concept of wireless client on-boarding was a hot topic at Wireless Field Day 5.  We had 3 AP vendors showing off their variations of how to bring clients onto the network securely: Aerohive, Motorola and Meru.

Just as these 3 vendors differ in how they approach the concept of wireless, their approach to client on-boarding varied quite a bit.  Let's briefly go through their solutions and talk about the highlights.

Monday, July 29, 2013

The Next Adventure: Wireless Field Day 5

So in just over 8 days I will be heading to San Jose for Wireless Field Day 5.  I've been a long time follower of the Tech Field Day events, it has been a great way for me to gain exposure to other parts of the industry that I don't get in my dayjob.  I'm very indebted to Stephen and my fellow delegates to inviting me to this event.  In addition to getting to meet some of the best professionals in the industry, I get to see solutions from a lot of different vendors and get to directly interact with them. Stephen and the Tech Field day super-crew have a pretty exciting line-up of sponsors for us to meet.

We have a 5 AP vendors coming:

Motorola and Aerohive have been WFD sponsors before, and I've gone back and reviewed their presentation.  Devin Akin donated some Aerohive lab gear to me last year and I've enjoyed putting it through its paces.  I'm pretty comfortable with their solution and can't wait to see what they have coming.  For Xirrus, Meru and Airtight, I've been digging through their product manuals and marketing material and have a laundry list of questions.  

The next category would be what I would call "Overlay Solutions."
  • 7Signal
    • Wireless Performance and Qualiity of Experience benchmarking system.
  • Fluke Networks
    • Spectrum overlay solution
7Signal as a company wasn't even on my radar 6 months ago.  From the material they have posted on the web, as well as some stuff sent to me by Veli-Pekka via Twitter, I hope to have a pretty good overview of the solution before heading to WFD.  Essentially it's on overlay solution that you can define performance and Quality of Experience (QoE) metrics in and their hardware actively associates and tracks the metrics across your wireless networks.  I've included Fluke here due to their overlay solution as well, but I primarily look at Fluke as a tools vendor.  Without knowing what Fluke intends to bring for Show-and-Tell, I'm including them in this category as well.

And last, but not least we have our wireless tools vendors. This is the like 3 tool vendors I want to see. 
  • Fluke Networks
    • Wireless Survey/Planning Tools, Troubleshooting Tools
  • Metageek
    • Spectrum Analysis and Visualization, Wireless Packet Analysis
  • Wildpackets
    • Packet Capture and Analysis

Some of the most valuable tools in my toolbag come from these vendors.   I use Planner/Surveyor as my design tool and it works for my needs. I have tested some of Fluke's other tools like their OneTouch AT. Even in the Beta, it was a tool with a lot of potential.  So there's lot of things I'm hoping to hear from the folks at Fluke.  Fellow WFD5 delegate Ryan Adzima has a post on their OneTough over at

I'm a pretty big fan of the Metageek hardware and software.  In addition to building some of the best Spectrum Visualization software in the market, they come from my hometown of Boise Idaho (Go Broncos!).  I utilize my Wi-Spy more than any other product in my toolbag by a long shot.

Omnipeek from Wildpackets is a solution I've looked at a number of times.  It's the gold standard for packet analysis from both a wired and wireless perspective and they have the solution that will make analysis work even as 802.11ac moves to speeds that will crush our USB 3.0 buses.

Overall, I am really excited to have the opportunity to join the rest of the delegates.  There is a small amount of nervousness surrounding this event for me personally.  For starters, I'm new to the TFD family.  I have met around half of the other delegates at other industry events.  I'm not an industry veteran, having just celebrated my 4th year in the networking industry (hey now, no young-in jokes).  And let's face it, there isn't a CCIE/CWNE number after my name.  But I do have a passion for wireless, networking, and really enjoy the opportunity to participate in the dialog at an event like WFD5.

I hope everyone logs in and participates live using the #WFD5 hashtag on twitter.  I've been asking questions using this method since I started following the delegates on Twitter during TFD3 , and have had delegates ask my questions to the sponsors on more than a few occasions.  You can watch live at

Sunday, July 21, 2013

CCIE-W: QoS Study Notes

I've had some people suggest that I should post more of my study notes.  QoS is one of those black magic topics that I really do need to understand more of.  I went through with one of my co-workers who gave me the Catalyst 3750 QoS in 30 lecture.  Between that and some of Jerome Henry's QoS videos on youtube, here the basics on QoS.  So here are my notes in all their ugly glory.

Sunday, July 14, 2013

One week to the CCIE-W lab, An approach

So in just over a week I will make my way to lovely San Jose to take my first attempt at the CCIE Wireless lab.  Sure it's a long shot passing on a first attempt, but even a failure at this point will give me the experience of "The Lab" and help point my studies in the right direction.  Not to say that I'm not going to give it my all, but I also don't "expect" to pass.

I'm currently going through the Fastlane lab material, and a lot of configuration guides and implementing them in my home lab.  I feel ok about the material.  I wish I had more time to to cover topics like MSE and WIPS, but I decided to go ahead with the lab on a short timetable.  At this point I'm looking more at SWTs (Stupid Wireless Tricks), learning some lesser known CLI commands and building a strategy for dealing with problems in the lab.

My focus for skills building is moving away from the simple "How Do I?" to more of a "How Do I Verify?" or "What is the quickest way to configure feature <x>?"

CCIE Wireless Study: WLC CLI commands:

This post is to be just a list of some useful CLI commands for attempting the CCIE Wireless lab.
Most of these commands are CLI only, or more useful from the CLI than from the GUI.

Tuesday, July 9, 2013

5760 Session Timeout: To Infinity and beyond!

If you're familiar with the advanced tab while configuring a WLAN  on a Cisco (Airespace) WLC, you've probably seen the Session Timeout checkbox and corresponding timer value.
This timer is used to setup how long a wireless user can remain connected before reauthentication is required.
While deploying a 5760 to match an existing 4400 controller, I attempted to disable the session timeout, but to no avail.  Let's look at my simple WLAN I have configured:

Saturday, June 1, 2013

How I Study: A cloud-enabled strategy to gain understanding

As a wireless engineer, I try really hard to embrace Mobility.  Even in my studies, Mobility is something I focus on.  I felt it was time to describe my self-learning process of study after showing some engineers at work some of the ways I use mobility to enhance my study process.  The problem with learning is that you need to do things to reinforce what you are learning.

“I hear and I forget. I see and I remember. I do and I understand." -Confucius

The point of studying and taking certs is to raise your understanding in a particular technology set.  Since this is self study, you can replace "hear" with "read."  Remembering is not the goal, but an understanding of the technology and how it fits into networking as a whole.  Adding mobility to this really stretches most study methods.

Thursday, April 11, 2013

Cisco Prime Infrastructure: Bug ID CSCud39395 Logins not Processed

A couple weeks ago I ran into an issue with Prime Infrastructure 1.2 where it was not responding to a login request.  It just sat there working on the login request but would never respond with a success or failure.

After stopping the NCS software, it would not start again.  I was presented with a failure and told to check the launchout.log

So I performed the Backup-logs and untar'd the file.  Here is what I was receiving in the file.

Starting Health Monitor as a primary
Checking for Port 8082 availability... OK
truststore used is /opt/CSCOlumos/conf/truststore
truststore used is /opt/CSCOlumos/conf/truststore
Starting Health Montior Web Server...
Health Monitor Web Server Started.
Starting Health Monitor Server...
Health Monitor Server Started.
Starting Remoting Service: Reporting Server
Checking for running servers.
00:00 Check complete. No servers running.
Starting Server ... 
 Start failed. Initiating shutdown. Please check logs/Startup.log.

Sunday, March 17, 2013

Powershell Module to Backup configs

One of my FAVORITE tools is powershell.  It's quick, easy to write and easy to reuse.
The esteemed Mr @BlakeKrone was asking for a script to backup configs off of a Cisco Switch.

To use this module, you have to run over and grab SharpSSH, a .Net implementation of JSCH.  I've used this in projects as far back as 2008.  It's a great library and very easy to implement in C#, or Powershell.  Just throw it in the same folder with the module and update the path in the script.

Tuesday, March 12, 2013

Building of my CCIE Lab

For those who don't know, I've been building my home Wireless lab for the better part of 2 years.  It started off as a 4402 load to me from a customer who had traded it in on an upgrade to a 5508.  Between some gear off ebay, some loaners from coworkers and clients alike it has grown.  In the last part of 2012, my employer purchased some equipment for lab up and demo purposes which currently resides in the lab.  I'm very thankful to my employer for sponsoring this effort, and various other individuals who have donated equipment.

Below is a list of what the lab consists of today, I'll amend the post as thing change dramatically.

Network Equipment
1x Cisco 2811 running CME
1x Cisco 3550 24 POE running IP Services
1x Cisco 3560cg 8 POE running IP Base
1x Cisco 2504-5 WLC
1x Cisco 2106 WLC
1x Cisco 5508-12 WLC
1x Cisco 881W

Access Points
1x Cisco 3602i
1x Cisco 1262
2x Cisco 3502i
1x Cisco 1142
3x Cisco 1242
1x Cisco 1042
2x Cisco 1131

1x Dell Precision Workstation T5500:  Quad Core, 30GB of Ram,
   - 2x 512GB Drives
   - 2x 250GB Drives
   - ESXi 5.1

1x Cisco ACS 5.2 VM
1x Cisco WCS 7.0 VM
1x Cisco Prime Infrastructure 1.2 VM
1x Cisco MSE 7.4 VM
1x Cisco vWLC 7.4 VM
1x Cisco ISE 1.1.2 VM
1x MS 2008r2 Domain Controller

2x Windows 7 x64 Clients VM
1x WinXP x86 Clients VM
1x Backtrack Linux 5r3
1x GNS3 Workbench
1x Apple TV Gen2

10 varied USB wireless adapters
 - Dynamically mapped to wireless clients via ESXi

1x Cyclades TS-3000 Terminal Server for OOB management

1x Windows 7 x64 Management Station (VM)

A very special thanks to @DevinAkin and @Aerohive for their donations, which not specifically listed in the "lab" equipment are in use, well tested and utilized.